I Can't Believe I'm Complaining About Long Threads, but Here we are.

October 29, 2012 9:36 PM

The Hurricane Sandy thread is crashing my browser on my home computer.

So are all the election threads, etc. Those don't bother me, because I know eventually I will be able to see them at my office - however, on this thread in particular, I can SEE that there is a lot of discussion about a major event that has impacted a lot of us, but I can't load up ANY of it, and I am concerned that by the time I see the thread later on it will be too late. Is this a "get a better computer" issue, or is there something that can be done?

posted by Curious Artificer to MetaFilter-Related at 9:36 PM (61 comments total)

At the very least, can anyone summarize?
posted by Curious Artificer at 9:39 PM on October 29, 2012

What browser and OS?

Did you install any add-ons? If so, try disabling some?
posted by slater at 9:45 PM on October 29, 2012

Well I think for a lot of us we read the big thread either by loading them once and then using the little refresh-o widget, or just clicking through to read it along with other threads in recent activity. This month or two has been remarkable because we've had the debates (and single threads devoted to them) and a few really long political threads. Up til just recently these threads were real outliers. Lately they haven't been. However we're feeling that after the election season we'll have to see if things get back to normal or not. For right now we don't have any plans to change the way large threads get dealt with [pagination and other options have some serous downsides that we feel would really alter the way conversations happen around here] but we'd be open to discussing ideas.
posted by jessamyn (staff) at 9:45 PM on October 29, 2012

Sort-of-summary: People are updating with information about what is going on for them, what it sounds like & looks like where they are. Some people are posting links to information about what is going on; for example, some hospitals in NYC are being evacuated. I don't think there is anything essential that you are missing. You could try watching the #sandy tag on twitter for something sort of similar, but not by mefites.
posted by insectosaurus at 9:50 PM on October 29, 2012

I'm still using Windows XP, and the issue is that when a thread gets over around 2000 comments it completely freezes the browser up - so, I CAN'T load it once and refresh it, because it never loads the first time.

Thanks for the mega-summary, insectosaurus.
posted by Curious Artificer at 9:51 PM on October 29, 2012

One option to try is turning off inline YouTube videos in your site preferences. Those take a while to process when the page loads and turning that off can save you a little time on long threads with over 2,000 comments. But since that will likely only save you a second at most, that's probably not the problem. I'm guessing your browser can't load a page that large. (It's 629 KB as I type this.) Definitely make sure you're not running any browser extensions like Greasemonkey, AdBlock, etc. All of those add processing time. If you have another browser handy that you don't use very often, you might see if you can load the thread there.

If you use a newsreader like Google Reader, you could try subscribing to the thread feed. (Here's the hurricane thread feed url.) That won't keep you up-to-the-minute, but in many readers you'll load the text incrementally without loading it all at once. So that might also work better on a system that can't load the entire page at once.
posted by pb (staff) at 9:56 PM on October 29, 2012 [3 favorites]

Wild. I use what I think is a first-gen iPod touch. No crashes. Colour me impressed.
posted by five fresh fish at 10:13 PM on October 29, 2012

oh, another option to try is logging out. Members get extra code that's required for adding favorites, comments, flags, tags, inline videos, and all sorts of extras. Non-members get a more streamlined page overall (though you'll see ads). So there's a chance your browser can handle that, but not the full version for members.
posted by pb (staff) at 10:19 PM on October 29, 2012 [1 favorite]

another option to try is logging out

NEVER!
posted by grouse at 10:20 PM on October 29, 2012 [11 favorites]

The Hurricane Sandy thread is crashing my browser on my home computer.

Hey, you're getting off easy. Hurricane Sandy herself blew up a ConEd power station in NYC!
posted by flapjax at midnite at 10:21 PM on October 29, 2012 [8 favorites]

It wasn't actually the hurricane that did that. The thing that happened was that somebody down the block from the power station logged onto Metafilter with the inline video option on and tried to get that thread to load. The power surge was just too much for the station to handle.
posted by koeselitz at 10:40 PM on October 29, 2012 [9 favorites]

I'm sure this is probably ground that's been covered before, but perhaps a good solution would be to also serve a minimalist version of a post with everything but text (or maybe text+basic html). I'm thinking something like metafilter.com/postnumber/postname?lofi. Is that practical to do?
posted by nerdinexile at 10:41 PM on October 29, 2012 [1 favorite]

another option to try is logging out

You can log out?!?

i kid
posted by Miko at 10:45 PM on October 29, 2012 [1 favorite]

Is that practical to do?

I'm not sure this problem is widespread enough to make the ongoing maintenance of another view worthwhile. I'm not even sure which browser Curious Artificer is using, and whether this is happenstance with the current state of their computer or something reproducible everywhere. So I'm not ready to cure this (so far) very narrow problem that someone is experiencing with a new feature.

We've had many, many conversations about the hassles of 1,000+ comment threads. The crashing browser problem is one aspect of these threads that we could address—but I don't think we're at the decision-making point yet.
posted by pb (staff) at 10:48 PM on October 29, 2012

Huh. Has Lofi Metafilter ever worked for anything beyond the front page? I guess probably not; even back in 2008 it was a non-maintained option anyway.

But that made me think of another, maybe more extreme, measure that Curious Artificer could take:

If threads really, really aren't loading on your home computer, you could always try the mobile site. The menu will be huge and blocky, but other than that it seems like it's pretty usable on a desktop – you can read threads and comment, all in kind of a lofi way. Much less load time. The huge threads seem to load a bit faster for me.
posted by koeselitz at 10:59 PM on October 29, 2012

I've lost power on and off throughout the day and night. During those times, that thread has been the best and most comprehensive source of accurate information I can access for what local conditions are, outside of weather underground. That includes major events like the Con Ed transformer fire and NYU evacuation. They've also helped debunk actual news reports from the major networks, such as the Oyster Creek Nuclear Power Plant story. They're helping us be supportive of and reassuring to one another during a very stressful crisis, and I think that's really invaluable.

But the thread's now at a point where it's taking a very long time to load on my HTC Incredible 2 using the mobile stylesheet. That's not a particularly old Android phone, either. Verizon only just stopped selling them in April.

Eventually the thread is going to become impossible to manage. Most likely within the day. So if I lose power, I'll lose access to an immediate resource. Forget posting there -- it's already taking more than a minute to comment and reload the thread on my phone. I realize these threads are outliers. And I recognize that all of the things I'm mentioning are not part of Metafilter's core mission. But they still exist. I'm positive I'm not the only person here who's finding that thread a vital source of up-to-date information during an emergency.

I really wish there were a mechanism in place that would allow us to break the thread into two parts, or that y'all would allow a new thread to be launched when a thread like it breaks 2000-2500 comments. We're almost at 3000 now.
posted by zarq at 11:04 PM on October 29, 2012 [10 favorites]

Is there a way it can archive itself as it goes? Like once it hits 1000 comments, the first 500 comments are stored elsewhere, collapsed sort of, so what you're left is a link that you can click if you want? (Facebook style, actually, I just realized).

Also, I have no idea what I'm talking about. all this is a spooky kind of magic to me.
posted by Miko at 11:12 PM on October 29, 2012

Here is a link to a zipped plain text version of the thread (hopefully this is okay). It's almost 900KB, but an advanced text editor (e.g. Notepad++) will handle it no problem. Obviously, this doesn't include the links, which are probably the most valuable content. (It also doesn't include the italics, so you can't see what's a quote, not that the thread is heavy with people quoting each other and debating.) On the bright side, the word count is well over 100,000 (subtracting the standard end-of-post user name and date text), so that's two NaNoWriMos in the bag and it's not even November yet.
posted by Homeboy Trouble at 12:04 AM on October 30, 2012

In case it helps some people to access the page (read-only), here's the coral cache address: http://www.metafilter.com.nyud.net/121309/Hurricane-Sandy.
posted by taz (staff) at 12:59 AM on October 30, 2012

... aaaand now it's down for some reason.

Unrelated, except that they may both possibly be casualties of the storm: it looks like the entire gawker network is down.
posted by taz (staff) at 1:14 AM on October 30, 2012

Gawker was already having trouble in the evening ET it seems, and they're not alone.
posted by goodnewsfortheinsane (staff) at 1:43 AM on October 30, 2012

If anyone's interested in some good ol' fashioned bad taste apocalypse journalism, look no further than this National Post article, where they follow a picture of the scene in Toronto where a woman was killed by a falling big box sign with a shot of people making out by candlelight in a New York restaurant:

Sandy leaves trail of destruction as Ontario braces for superstorm’s worst
posted by mannequito at 2:38 AM on October 30, 2012

At the very least, can anyone summarize?

Okily.
posted by Wordshore at 2:41 AM on October 30, 2012

fwiw, I loaded the thread after there were over 2k comments. And that was the first time I had loaded it.

While slow, it worked with no problem.

Chrome (latest version), lots or RAM and HD space for cache. Win 7
posted by lampshade at 3:13 AM on October 30, 2012

Don't log out, just go Private Mode. Ctrl-Shift-P (Firefox) / Ctrl-Shift-N (Chrome, Opera) and go.
posted by mahershalal at 4:31 AM on October 30, 2012

Ahhh....for you folks that try the mobile site... I found I had to log out and log back in to get OUT of the mobile site
posted by HuronBob at 5:24 AM on October 30, 2012

MetaFilter: Also, I have no idea what I'm talking about
posted by Egg Shen at 5:37 AM on October 30, 2012 [1 favorite]

There's a toggle way at the bottom of the mobile site, HuronBob.
posted by goodnewsfortheinsane (staff) at 5:46 AM on October 30, 2012 [1 favorite]

I was wondering if it was time for a metatalk for people to check in that they're okay, because that longboat is crashing my HTC Incredible's browser now too, and I expect a lot of East Coast people checking in may just have phone access and a fast-loading MeTa might be better for checkins at this point? (We could call it the rowboat thread!)

I'm still reading along in the main thread on my computer, but it has surpassed my phone's abilities (and my phone did pretty well with those politics longboats).
posted by Eyebrows McGee at 6:10 AM on October 30, 2012 [1 favorite]

For what it's worth, even though the FPP loads quickly on my computer, I can't access most news sites - I think they're overloaded, as every non-news site I try loads just fine for me. If that's true for others, the FPP is the best source of information for many people.

Perhaps, as a one time exception, there could be a new FPP? I'm sure it will move quickly, but at least it would help for now.
posted by insectosaurus at 6:33 AM on October 30, 2012

A check-in MeTa thread sounds good at this point Eyebrows McGee, if you want to post it, go right ahead.
posted by goodnewsfortheinsane (staff) at 6:39 AM on October 30, 2012 [1 favorite]

Done.
posted by Eyebrows McGee at 7:31 AM on October 30, 2012 [1 favorite]

pb -- have you done much profiling of MeFi's html/css/javascript rendering? If not, I'm wondering if there might be some random tweaks that would improve rendering performance.

I've been wondering about this for a while, actually, so I finally decided to take a quick stab at it. I downloaded a local copy of a thread, gave it 10,000 comments, removed some calls to remote scripts and added a timer to the top and bottom of the page. On my machine a thread that size takes about three seconds to render once it's downloaded. It seems like most of the three seconds just goes to parsing and applying CSS to all those elements. So in my initial tests rendering seemed to go about 10% faster if I took out a few unnecessary tags in the comments (empty spans, and <br> tags between comment divs). I could also save close to 10% by removing all the inline CSS styles (although I didn't replace them with global styles, so I'm not sure how much would actually be saved). That seems promising enough to merit more experimentation.

So, I'm not really an expert at cross-browser optimization, and maybe you already have things carefully tweaked for purposes I wouldn't know about. But if you'd be interested in optimization help, this seems like the kind of code-golf game that some Mefites might have fun with -- maybe with a clean test-bed reference file to download and play with?
posted by jhc at 9:56 AM on October 30, 2012 [1 favorite]

have you done much profiling of MeFi's html/css/javascript rendering?

Yeah, we have done quite a bit of work on that. There's always more that we could do, and some at-home experimenting is fine. If you find a tweak that can speed things up we can definitely take a look at it. I don't think we need to set up a special place for development on this.

Keep in mind that the MeFi HTML has evolved slowly over the years. But one nice aspect is that it has been battle-tested and tweaked in just about every browser/OS imaginable. There is some cruft, absolutely, but it works for a lot of people as-is on all sorts of unique setups. So one person's unnecessary <br> tag is another person's necessary <br> tag.

Again, I sympathize with the problem Curious Artificer is having. And the problems a couple people have mentioned having on their phones. Other people with mobile devices are doing fine. And the vast majority of people on a desktop are doing fine. So I don't think it's time to rally the troops for a technical solution. I'd like to know if Curious Artificer is having a problem that's specific to a certain browser—that might be the answer to this specific problem, I don't know.
posted by pb (staff) at 10:24 AM on October 30, 2012

pb, I'm using Internet Explorer here and at home - here, I have no problem with any of the long threads, but they routinely freeze up the machine at home. I'll check which version I've got there tonight. In neither case do I use any add-ons, ad blockers, etc.
posted by Curious Artificer at 11:43 AM on October 30, 2012

Thanks for the info. Yeah, knowing the versions you're using will help me know where to test.
posted by pb (staff) at 11:46 AM on October 30, 2012

Miko: "You can log out?!?"

But you can never leave.

Blah blah earworm blah.
posted by deborah at 12:02 PM on October 30, 2012 [1 favorite]

pb: "And the problems a couple people have mentioned having on their phones. Other people with mobile devices are doing fine."

I have complete and utter faith in your abilities pb but this sort of reads like "there's only been a few complaining" - I would caution that not everyone with a problem takes the initiative to complain or is brave enough to post a meta.

These threads kill my mobile browsers (dolphin and stock) on my droid. And I gave up completely trying to comment with my phone. Just adding another data point.

I'll try some of the tricks above (i.e. turning off inline youtube) but it feels like when it gets to the point where you have to turn off features it indicates a bigger problem. Sort of like not being able to run the a/c in a crappy car when it's hot outside because it will overheat.
posted by Big_B at 1:03 PM on October 30, 2012 [1 favorite]

Is there a way it can archive itself as it goes? Like once it hits 1000 comments, the first 500 comments are stored elsewhere, collapsed sort of, so what you're left is a link that you can click if you want?

I was going to suggest something like this, but out of 1000, e.g., I would hide the middle 500 instead of the first or last 500. It would take some AJAXy code to pull comments within a certain range, either by time or by comment number (I don't know if deleted comments stay in the database).

I've never had any issues with crashing, but the rendering time is noticeable on my computer, iPhone, and iPad. Plus I feel bad for wasting server resources if the inline updater doesn't work because it was in a background tab on the iPad and I have to reload the whole thing.
posted by stopgap at 1:25 PM on October 30, 2012

Actually, I take it back. Even hiding the middle block of comments would break things like searching within the thread and direct links to comments. Would using display: hidden in the CSS for a big chunk of comments improve rendering times? I think the timestamp links to comments would still work in that case.
posted by stopgap at 1:38 PM on October 30, 2012

Internet Explorer is a horrible browser. Maybe IE9 deserves praise but it loads extremely slow on my i5 laptop compared to Google Chrome. The first thing I do to "speed up" an old slow browsing PC for friends or family is install Chrome. Unless it's infested with malware, anyway! Then Chrome comes in slightly later. It makes a huge difference.
posted by lordaych at 3:15 PM on October 30, 2012 [1 favorite]

How do people feel about reading MeFi using a favorites metric (like the MFCF bookmarklet)? I confess I read most threads at at least 1+, longer ones at higher filters (a lot of days I only get here once a day, though*). Of course, there isn't perfect jibing between favorite-ness and utility, but it seems far more likely to be widely useful than just performing a middle-ectomy.

* I suspect this is why I am so high on the actual-last-comment-in-the-thread ranking somewhere in the stats.
posted by dhartung at 4:47 PM on October 30, 2012

The hurricane thread is starting to glitch for me. Sometimes when I hit the "new comments" button it doesn't load all the comments in the hurricane thread. Here's what it looks like. The top image is unglitched, b1tr0t's comment is followed by Sticherbeast's comment. However, in the bottom image, b1tr0t's comment is followed by lyra4's comment, and the 30 or so comments in between are missing.

I'm not entirely sure it's happened before, but there have been times where I've hit the "new comments" button and it hasn't returned as many comments as I was expecting. But I've always chalked it up to deletions or me being silly.

I'm using Google Chrome on Windows 7.
posted by Kattullus at 5:47 PM on October 30, 2012

Sorry Kattullus, not sure what happened there and there isn't a way to go back and check. We're definitely in outlier territory with these epic threads. It's possible that you clicked "show" and the process failed for some reason—but a new "x new comments, show" message appeared, missing those 30 comments. I'm not sure.
posted by pb (staff) at 6:10 PM on October 30, 2012

If you're ever in doubt, reload. And let the server grind away on it. You'll get the official view that way every time.
posted by pb (staff) at 6:16 PM on October 30, 2012

I still have both these tabs open and haven't touched them since noticing the glitch. If I save the source and e-mail to you, could that tell you anything?
posted by Kattullus at 6:16 PM on October 30, 2012

No, not really. The source is the source at the time you loaded the page. While the page is polling in the background and you're adding in new comments, the state is changing within the browser. I don't have access to that state. And even then, it wouldn't tell me exactly what misfired.

Maybe a network glitch? Maybe a JavaScript variable was off somewhere? I appreciate it, but until I can reproduce it there are too many variables to know.
posted by pb (staff) at 6:20 PM on October 30, 2012

Ah okay. I have no scripts running, incidentally. If I ever notice it again I'll let you know.
posted by Kattullus at 6:29 PM on October 30, 2012

When we lost power yesterday in Arlington VA I was trying to get information through the thread on my phone. And I did, for which I was very grateful! I even posted! But it took me so long to do all that (posting took forever, as the letters I typed didn't show up in the comment box until 15 seconds later) that I burned 40% of my phone battery, that I could not then recharge until late this morning when a neighbor ran us an extension cord.

I know it's not MetaFilter's job to spend money so that we can all comment to one another from our phones during the latest emergency. Maybe that's what social media like twitter and Facebook are for? I don't know. Metafilter and my local community online journal do serve that purpose for me. But I probably can't really come here in an emergency anymore if it takes so long to get in and out that it eats half my phone battery. I don't know what the solution is. But I do appreciate the info in these threads during emergencies. I remember reading the 9/11 thread from my office in DC across from the FBI building, before I was even an official member, and being really grateful to have the info and links and personal stories there in real time.

I guess what I'm saying is that I'd like to spend the apocalypse with you, MetaFilter, if you could just go a little easier on my phone battery.
posted by onlyconnect at 6:59 PM on October 30, 2012

One design I've seen used on other sites/apps is like this:

Recent Activity says: "3551 total comments. 1414 since your most recent comment, last 10 shown below..."
That "most recent comment" link goes to a page that only includes the FPP and the new posts.
Between the FPP and the new content is a link that says "show all posts" and the reloads the page.

This way you don't have pagination and all that junk, but you do only download the new content.
posted by smackfu at 7:07 AM on October 31, 2012

Pb - Not sure if you're still watching this thread, but I noticed in the Sandy thread that on reload, my browser will only jump to my most recent comment, not the last comment I've loaded. Isn't it supposed to go to last comment loaded? I'm using the safari browser on an updated ipad3.
posted by smalls at 8:31 AM on October 31, 2012

smalls, what link are clicking to get into the thread? My guess is that you have your most recent comment targeted in the URL. (Look for a #[comment_id] in the URL.) If you remove that, it won't target any specific comment—and your browser should go to approximately where you were reading when you reload.
posted by pb (staff) at 8:45 AM on October 31, 2012

Sort of like not being able to run the a/c in a crappy car when it's hot outside because it will overheat.

So what you are proposing is to fix the Sun instead of fixing the car.
posted by banshee at 10:06 AM on October 31, 2012 [1 favorite]

My data point on Nexus 7 tablet - at around 2000 comments, it was taking around 90 secs to load. After 3000 it just crashed the browser. :-(

Also, is there a way to convert the post box into a native text box without JavaScript? On these large threads, the page reflows caused by the preview update just kill performance. Additionally, the built-in spellcheck and cut paste logic fight with whatever js in there.

Thanks for the free ponies!
posted by PissOnYourParade at 10:52 AM on October 31, 2012

When I took out the #comment, I ended up at the beginning of the thread. Maybe I did something weird?
posted by smalls at 1:34 PM on October 31, 2012

Maybe I did something weird?

Nope, that's expected. Now going forward, when you reload the page, it should take you to where you were reading. If that doesn't work for you then start targeting comments by clicking on the timestamp. Each time you click a timestamp, the targeted comment will change. Then when you reload, the browser will bring up the targeted comment. You can also look for the yellow triangle on the side, that will appear if you have a comment targeted.
posted by pb (staff) at 1:59 PM on October 31, 2012

I see he's still here, but pb says he was able to duplicate the problems I was having with my browser/OS setup, and has tweaked some sort of JavaScript doohickey that may help the load time.

As always, our mods are the best mods!
posted by Curious Artificer at 2:08 PM on October 31, 2012 [1 favorite]

Yeah, I was just about to post that. We're going to go into "less nice" mode on threads that go over 1,000 comments. Technically, the problem older versions of IE had was related to looping through the DOM to clean up some things after the page loads. And once there are over 1,000 comments that's a lot of HTML and links to scan through.

So going forward we're skipping a couple nice extras on long threads so that the page will load a little faster for everyone. The first thing to go is removing the JavaScript Required links if JavaScript is enabled. We pull those out (using JavaScript) if you do in fact have it enabled, but that takes processing time. No functionality will change. Next, if you have the profile preference "Open links in new windows?" checked, long threads will now open all links in a new window or tab, including links within that page. That's a hassle—especially for epic threads because they take a while to load—but we felt like the tradeoff was worthwhile.

Those two changes alone drastically reduced the page load time for older browsers, and it should help with mobile devices as well.
posted by pb (staff) at 2:18 PM on October 31, 2012 [3 favorites]

PissOnYourParade, yeah, you have a couple options. You can enabled the mobile view for your device by visiting mobile.metafilter.com. That will turn on the mobile view which turns off Live Preview during commenting.

You can also explicitly disable Live Preview by clicking the "Hide" link where it says "Live Preview". That will set a cookie that tells the site not to enable Live Preview, but it will force you to use the full HTML preview before you post. So that may or may not work for very long threads depending on how those are performing for you on your tablet.
posted by pb (staff) at 2:29 PM on October 31, 2012 [1 favorite]

I've been getting a lot of JRun errors today. I don't know this is in any way related to the monster threads (I've seen it most often when I refresh the election thread, but also for other pages that aren't monster threads).
posted by Kattullus at 10:44 AM on November 6, 2012

Thanks for the heads up. I've seen that once today too. I think it is the extra traffic in the election thread. The server seems to be recovering quickly from high loads, but yeah, JRun errors mean the server is extremely busy.
posted by pb (staff) at 1:29 PM on November 6, 2012