Reducing the load time for long threads.

March 14, 2002 7:50 AM

For exceptionally long threads, would it be more CPU-friendly to stop rendering new pages every time someone clicked into the page after a certain point? Say, arbitrarily, 100 posts?
posted by Hildago to Feature Requests at 7:50 AM (7 comments total)

Instead, maybe it could load a cached page that gets made every 2-3 minutes?

And sorry to pick on two of your threads Miguel, they were just the first huge threads I could find.
posted by Hildago at 7:52 AM on March 14, 2002

I was thinking there should also be a limiter on the "view my comments" feature, listing 50 at a time. Matt has mentioned in the past that this is a query-intensive feature.
posted by werty at 8:02 AM on March 14, 2002

There's very little reason that every thread couldn't be cached as static HTML, with just the comment box being inserted by a script (since its content varies depending on who's logged in). Simply update the cache (which could be a flat file, or kept in RAM) whenever someone posts. This would eliminate virtually all of the database queries for displaying threads. The database would then be primarily for searching. (If it weren't for the searching, MeFi would almost not need a database backend at all.)

However, Matt knows the MeFi code better than I, it could well be that this would be a significant amount of work which he simply doesn't have time for at the moment. I'm sure he's considered something like this, though -- I would.

Another possibility would be to stick a caching proxy such as Squid in front of the main Web server. I think it should be possible to set it up so it only caches pages for people who aren't logged in (keying off the cookies in the header), and have the cached pages automatically expire every 5 minutes or so. This would be somewhat less work, I think, and should significantly reduce the load on the server, but I've never done it so it might be more work than I think.
posted by kindall at 8:42 AM on March 14, 2002

Free Republic made this switch a while back, to great success both in terms of processor load and end-user suffering. (Oh, and wasted bandwidth, too.) It lists 50 at a time as a default, though you can adjust it to 20, 100 or 250 if you so desire.
posted by aaron at 8:42 AM on March 14, 2002

I'm working on a flat file system right now.
posted by mathowie (staff) at 10:30 AM on March 14, 2002

Sweeet.

I figured that pagination, the way an app I'm building at work does it, would be even better, though.
posted by SpecialK at 8:20 PM on March 14, 2002

How do you get CF (or, in the case I'm actually interested in PHP) to render static pages rather than generating them on the fly? I mean directly, as opposed to using a proxy to do caching.
posted by rodii at 5:52 AM on March 15, 2002