How to reduce server load on large threads

February 27, 2004 10:46 AM

Okay, since I sort of wasted my last MeTa post by being too fast on the draw, how about something more constructive. I think we could save a great deal of overhead on the server by changing the way 100+ post threads are displayed.
posted by y6y6y6 to feature requests at 10:46 AM (29 comments total)

With the election coming up, and Bush willing to make gays vs religion part of the campaign, I think we'll be seeing a great many threads go over 200 posts. With the server already throwing JRun errors and timeouts as often as not, perhaps we should change the way these long threads get displayed.

Personally I don't need to see more than the last 20 or so posts. But every time I refresh or post a comment the server has to hold in memory and send all 300. Which is about a third of a meg per pop.

Could we make it so that after a thread goes over 100 posts it will only display the number shown in the "(16 new)" link when that's clicked on? It could still display the full list if the "419 comments" link was clicked.

For people who aren't logged in you could add a "Last 20" link so that non-members could also help limit server load.
posted by y6y6y6 at 10:46 AM on February 27, 2004

I think that makes sense, as the longer threads will also be receiving more calls anyway. I'd like to see more than the last 20 comments tho (perhaps this could be set in the preferences).

Is there anyway we (i.e. someone else) could figure out how much bandwidth each thread has sucked up, and correlate this with thread length? There's probably a fat-tailed distribution there, with a small number of long threads accounting for a lot of the traffic. Acting on the long threads could significantly cut traffic, if Matt is worried about that.
posted by carter at 11:14 AM on February 27, 2004

I've been thinking of paginating threads every 50 or so comments.
posted by mathowie at 11:33 AM on February 27, 2004

Paginating is a very good idea. Often after a certain number of posts have been made, some conversations begin repeating themselves. At one time it's lazy to comment on a thread when one hasn't read the whole thing, but who wants to sort through 100+ posts? Especially our at-work-brethern? Reading the first 20 to 50 should give people a jist of what is happening in the whole thread, so paginating may be helpfull. I do wonder, however, will it be easier for people to screw around and troll if threads are broken into pages?

Oh, and a funny post I almost put up on the front page, but didn't want to post the third such thread in three days.
posted by elwoodwiles at 11:53 AM on February 27, 2004

OH, COME ONE, ELWOODWILES!
4 out of 5 mefites want a front page link that says "Gay Sex On Uranus?"
posted by quonsar at 12:28 PM on February 27, 2004

come one. heh.
posted by quonsar at 12:30 PM on February 27, 2004

Ok, done.
posted by elwoodwiles at 12:39 PM on February 27, 2004

Boo to paginating. I don't mind idly scrolling through a thread to read 300 comments or so, but I hate seeing 'Page 1 of 6'. You end up with people just reading the first 20 posts, and the commenting, missing the majority of what's been said already.

If you do end up paginating, please please please have a 'view all comments' link. As lazy as this sounds, I really hate having to click a dozen times, waiting for the page to load a dozen times, etc. I really like the simplicity that is MeFi's comments system.
posted by Jairus at 12:44 PM on February 27, 2004

I also strongly dislike pagination, but unless someone can come up with a better solution that addresses the server load problem, it's probably necessary to grin and bear it.
posted by rushmc at 1:41 PM on February 27, 2004

I agree with Jairus: I want to be able to see/search the whole thread at once.
posted by languagehat at 1:54 PM on February 27, 2004

I also strongly dislike pagination, but unless someone can come up with a better solution that addresses the server load problem, it's probably necessary to grin and bear it.

How about just closing the thread to comments once it reaches 100 (or whatever)? It's not like there'll never be another opportunity to comment on any particular hot topic.
posted by timeistight at 2:28 PM on February 27, 2004

I agree, no pagination please! It's one of the most important advantages of the Mefi format over just any old bulletin board software.

One server load improvement, though, could be counting new comments since last login entirely using javascript and cookies.

Instead of running a query like this...

SELECT COUNT(commentid) FROM comments WHERE timestamp > lastlogin

...for each FPP, you would run a JavaScript function like this pseudocode for each FPP....


function commentcount(currentcount, cookie_lastcount, id){
newcount = currentcount - cookie_lastcount;
if(newcount > 0){
commentjump = cookie_lastcount + 1;
document.write('<a href="/mefi/'+id+'#'+commentjump+'"');
document.write(newcount+' new');
document.write('</a>');
}
cookie_lastcount = currentcount;
}


The only drawback would be that browsers without Javascript wouldn't get a new comment count, but the reduction in database hits loading the front page would be significant (unless I'm assuming incorrectly about how the current setup works).
posted by 4easypayments at 2:33 PM on February 27, 2004

How about just closing the thread to comments once it reaches 100 (or whatever)?

how about not?
posted by quonsar at 3:40 PM PST on February 27

how about not?
posted by quonsar at 3:40 PM PST on February 27

how about not?
posted by quonsar at 3:40 PM PST on February 27

how about not?
posted by quonsar at 3:40 PM PST on February 27

how about not?
posted by quonsar at 3:40 PM PST on February 27

... etc etc

how about not?
posted by quonsar at 3:40 PM PST on February 27
posted by quonsar at 3:37 PM on February 27, 2004

Cutting off threads at a set number of comments will only motivate the more trollish among us to do "new" and more dubious posts on the same subject just to continue the discussion.

In other words...

what quonsar said.

what quonsar said.

what quonsar said.

what quonsar said.

what quonsar said.

what quonsar said.

what quonsar said.
posted by wendell at 4:45 PM on February 27, 2004

FIRST POST!

(or something like that)
posted by shepd at 4:59 PM on February 27, 2004

I think it would be more like:

3rd to the last post!

2nd to the last post!

Next to the last post!

LAST POST!
posted by mischief at 5:02 PM on February 27, 2004

No, *I'M* last post, SUCKAHS!
posted by shepd at 5:05 PM on February 27, 2004

Last Post
posted by inpHilltr8r at 5:39 PM on February 27, 2004

How about pagination begins the second time the user reads the thread? The first time, all comments are displayed on a single page. The second time, the reading begins with "X new comments" and goes backwards 50 comments at a time.
posted by Hildago at 5:50 PM on February 27, 2004

Pagination is silliness. If heavily-posted threads really are the source of most of the database load, why not just append each chunk of 20 comments to a file, include the static file, then crank out the remainder with the customary database hammering? Then no given pageload is more than 20 rows of resultset, we get to keep the same user experience, and pagination dies, dies, dies.

Not, of course, that Matt needs the peanut gallery to tell him how to code the site.
posted by majick at 9:08 PM on February 27, 2004

Sure he does, that's what metatalk is all about.
posted by ashbury at 9:39 PM on February 27, 2004

How about pagination begins the second time the user reads the thread?

That would require keeping track of which threads each user had read, which is not impossible, but which I doubt would really improve performance significantly.

Generating static HTML files for the bulk of the comment pages, rather than retrieving all the comments from the database each time, is the better choice, as majick says. Update that when a new post is made and have the actual CF script just include the posts and generate the form. Doing it in chunks of 20 is not necessary and more complicated.

Throwing more RAM at the machine seems to work, too, and may be a more efficient use of Matt's time. ;)
posted by kindall at 10:37 PM on February 27, 2004

I think we could save a great deal of overhead on the server by changing the way 100+ post threads are displayed.

As a web applications programmer, everytime the MeFi server goes down (which is often) I think "Of course".

Every client that accesses Metafilter demands so much from the server. They demand the 50 or so (on average)most previous FPPs on the default page. You click on a popular FPP. You demand 200 posts. It's no wonder the server overloads. The behaviour of users too:- we post, we check to see if people have responded. Do that with a popular thread and you will get so many requests, it's not funny. And do we read the first 180 posts of a 200 post thread each and every time? No, we skim them once. Why do we need them fed to us everytime we access a thread? You could have an area that summarises the latest posts to the FPPs you've recently posted to. Click on the link and it list the latest 15-20 posts (as has been mentioned many times above here).

Bad web-site design involves incredibly long pages that take ages to load - simple.

I don't know if these scripts are third-party and therefore cannot easily be changed, or whether it's bespoke coding (in which case, what's your excuse?).


That would require keeping track of which threads each user had read, which is not impossible, but which I doubt would really improve performance significantly.

It is possible, with cookies. Each cookie is unique to each thread. The cookie identifies how many times you've visited the thread. You access the thread a second time or more, it gives you the latest 20 posts (for example).
posted by SpaceCadet at 7:03 PM on February 28, 2004

MetaTalk : Backseat driving for web-monkeys
posted by inpHilltr8r at 7:54 PM on February 28, 2004

here's a great idea:

MAKE THE DAMN PAGINATION OPTIONAL

Now shut up and make me soup.
posted by angry modem at 7:43 AM on February 29, 2004

*boils angry modem into soup*

Oh, and optional pagination would be a good thing, because lots of people would love it but I (and others, it seems) would hate it.
posted by dg at 1:45 PM on February 29, 2004

*takes a sip of angry modem, adds salt*

Yeah, what dg said.
posted by languagehat at 6:14 PM on February 29, 2004

"Doing it in chunks of 20 is not necessary and more complicated."

That's true in a single user environment, or a low-traffic site which is effectively so. For a heavy site, you need to look a bit deeper and pick a pattern that fits better: There's no potential for contention for data 20 rows back like there is for the insert that put this comment where you see it. An occasional dump to cache is much easier and less complicated than implementing a lock-and-wait mechanism to write to the cache file. The tip of the thread is better served with database I/O, but the bulk of it is much better off being static.
posted by majick at 7:42 AM on March 1, 2004

Optional, customizable (10, 20, 50, 100 with some default) pagination?
posted by callmejay at 9:26 AM on March 1, 2004