Metafilter comments rated. September 29, 2009 9:12 AM Subscribe
Metafilter's comments are better than Youtube's. Now with graphs.
Headlines: Comments here have longer sentences, lengthier words and the better ones are properly punctuated. The best of all are very long.
Headlines: Comments here have longer sentences, lengthier words and the better ones are properly punctuated. The best of all are very long.
Eh - out of context, a one-line quip may look like a bad comment, but may be the most hilarious thing said in that thread.
That said, I'm still willing to bet that there's a strong correlation between number of favorites and comment length here on MeFi. Maybe I'll query the info dump when I get time later.
posted by chrisamiller at 9:17 AM on September 29, 2009
That said, I'm still willing to bet that there's a strong correlation between number of favorites and comment length here on MeFi. Maybe I'll query the info dump when I get time later.
posted by chrisamiller at 9:17 AM on September 29, 2009
You know, with the information here, I could write the perfect Metafilter sentence. Let me work on it a bit and please have your favorites ready.
posted by ricochet biscuit at 9:25 AM on September 29, 2009
posted by ricochet biscuit at 9:25 AM on September 29, 2009
Interestingly, I consider your crap, jackass dog is presumably a gay douchebag.
(You? My God...)
posted by Rhaomi at 9:26 AM on September 29, 2009
(You? My God...)
posted by Rhaomi at 9:26 AM on September 29, 2009
So now we have some douchebag giving jackasses a metric for their crap comments?
posted by robocop is bleeding at 9:27 AM on September 29, 2009 [6 favorites]
posted by robocop is bleeding at 9:27 AM on September 29, 2009 [6 favorites]
At least on websites with an audience that is not actively malevolent, longer comments seem to be better.
MetaFilter: not actively malevolent
posted by Joe Beese at 9:33 AM on September 29, 2009 [2 favorites]
MetaFilter: not actively malevolent
posted by Joe Beese at 9:33 AM on September 29, 2009 [2 favorites]
I really tried to skew the results by making a lot of really lame comments.
posted by It's Raining Florence Henderson at 9:42 AM on September 29, 2009
posted by It's Raining Florence Henderson at 9:42 AM on September 29, 2009
No controls. Fail.
posted by Blazecock Pileon at 9:42 AM on September 29, 2009 [1 favorite]
posted by Blazecock Pileon at 9:42 AM on September 29, 2009 [1 favorite]
Interesteringshly I am fascinabobulated by the zig-zagging risealation as comments traversify through a couple phases of succincter-woulda-been-a-betterer.
posted by fleacircus at 9:48 AM on September 29, 2009 [1 favorite]
posted by fleacircus at 9:48 AM on September 29, 2009 [1 favorite]
That said, I'm still willing to bet that there's a strong correlation between number of favorites and comment length here on MeFi. Maybe I'll query the info dump when I get time later.
There's no information about comment length in the Infodump, unfortunately. Although I wonder if this is something that could be added? I agree with not having the actual comment text in there, but it seems like comment length would be pretty harmless.
posted by FishBike at 9:54 AM on September 29, 2009
There's no information about comment length in the Infodump, unfortunately. Although I wonder if this is something that could be added? I agree with not having the actual comment text in there, but it seems like comment length would be pretty harmless.
posted by FishBike at 9:54 AM on September 29, 2009
Also, see his earlier studies on "Lies" and "Damn Lies".
posted by GuyZero at 9:56 AM on September 29, 2009
posted by GuyZero at 9:56 AM on September 29, 2009
MetaFilter: not actively malevolent
Speak for yourself.
posted by Cool Papa Bell at 10:29 AM on September 29, 2009 [2 favorites]
Speak for yourself.
posted by Cool Papa Bell at 10:29 AM on September 29, 2009 [2 favorites]
His conclusion, "Given this data, my recommendation is to not restrict comment length, or to restrict it at a high level", doesn't make a lot of sense to me. His data show that long comments are good for MetaFilter, but they also show that length is largely irrelevant to YouTube. Presumably 5000-character YouTube comments would be as inane as 20-character comments.
Or did I miss something?
posted by le morte de bea arthur at 10:30 AM on September 29, 2009
Or did I miss something?
posted by le morte de bea arthur at 10:30 AM on September 29, 2009
Well, he's certainly wrong about one thing: anyone who's ever been a fan of a certain 17th century treaty can tell you that Metafilter does, in fact, restrict comment length.
posted by dersins at 10:34 AM on September 29, 2009 [4 favorites]
posted by dersins at 10:34 AM on September 29, 2009 [4 favorites]
anyone who's ever been a fan of a certain 17th century treaty can tell you that Metafilter does, in fact, restrict comment length
Yeah, one of my only deleted comments was deleted for being too long. Apparently some of my contributions are of such a high quality that even MetaFilter can't contain them.
posted by burnmp3s at 10:44 AM on September 29, 2009
Yeah, one of my only deleted comments was deleted for being too long. Apparently some of my contributions are of such a high quality that even MetaFilter can't contain them.
posted by burnmp3s at 10:44 AM on September 29, 2009
> Or did I miss something?
You missed the several places in the post where he explains that YouTube has a firm 500-character limit.
posted by ardgedee at 10:53 AM on September 29, 2009
You missed the several places in the post where he explains that YouTube has a firm 500-character limit.
posted by ardgedee at 10:53 AM on September 29, 2009
.
posted by Kirth Gerson at 10:57 AM on September 29, 2009
posted by Kirth Gerson at 10:57 AM on September 29, 2009
The pasted-in Treaty of Wesphalia? Now that was a quality comment!
posted by Kirth Gerson at 10:58 AM on September 29, 2009
posted by Kirth Gerson at 10:58 AM on September 29, 2009
did he die?
posted by fire&wings at 11:10 AM on September 29, 2009
posted by fire&wings at 11:10 AM on September 29, 2009
Looking at MetaFilter comments, there seems to be a clear correlation between comment quality and comment length. At least on websites with an audience that is not actively malevolent, longer comments seem to be better
Mwah ha ha ha ha.... I don't think he knows some of us very well.
posted by MCMikeNamara at 11:13 AM on September 29, 2009
Mwah ha ha ha ha.... I don't think he knows some of us very well.
posted by MCMikeNamara at 11:13 AM on September 29, 2009
The Treaty of Westhphalia is not deleted for its length, but for its radical content, which cannot be unleashed on an unprepared audience. Oh, sure, superficially it seems a rather mundane document detailing the truce at the end of the Thirty Years War, but buried in it, encoded into the independence of Bremen section, is the key to unlocking the pathway to the Elder Gods and the eldrich horror therein, the breaching of which had bene the longterm goal of the House of Habsburg, and the supression of which was the true cause of the war. I risk so much just by telling you this; even now, the watchers scratch at my window, and a strange music plays just on the edge of my perception, written in no human scale and expressing an intelligence no human has ever possessed. THEY COME FOR ME! IA! IA!
posted by Astro Zombie at 11:13 AM on September 29, 2009 [14 favorites]
posted by Astro Zombie at 11:13 AM on September 29, 2009 [14 favorites]
You missed the several places in the post where he explains that YouTube has a firm 500-character limit.
I think le morte de bea arthur's point was that the data shows that once YouTube comments get beyond 10 or so characters, there is little or no improvement in their ratings, whereas MetaFilter comments show an upward trend well before hitting YouTube's limit.
Personally I think the claim that removing the 500 character limit on YouTube or similar comments would directly improve comment quality is dubious. Long and well-written comments are more of a side effect of having meaningful discussions than a fundamental result of having the ability to post long comments. There are many other factors, such as the userbase, moderation, and presentation of the comments (i.e. not reverse-chronological order with only a few per page) that probably keep comments at a poor level of quality on some sites regardless of technical length limits.
posted by burnmp3s at 11:19 AM on September 29, 2009
I think le morte de bea arthur's point was that the data shows that once YouTube comments get beyond 10 or so characters, there is little or no improvement in their ratings, whereas MetaFilter comments show an upward trend well before hitting YouTube's limit.
Personally I think the claim that removing the 500 character limit on YouTube or similar comments would directly improve comment quality is dubious. Long and well-written comments are more of a side effect of having meaningful discussions than a fundamental result of having the ability to post long comments. There are many other factors, such as the userbase, moderation, and presentation of the comments (i.e. not reverse-chronological order with only a few per page) that probably keep comments at a poor level of quality on some sites regardless of technical length limits.
posted by burnmp3s at 11:19 AM on September 29, 2009
A lot of my comments are very long, but they are also atrociously punctuated. It is actually very embarassing. The following are to blame:
(a) society,
(b) the small font,
(c) the excessive width of the comment box, and
(d) the cramped line spacing in the comment box.
If anyone can suggest a way to improve any of these items, particularly (a), I shall incorporate it directly into my comment workflow. You could then be assured that even my most logorrheal comments would include grammatical apostrophes rather than catastrophes.
posted by Pastabagel at 11:22 AM on September 29, 2009 [1 favorite]
(a) society,
(b) the small font,
(c) the excessive width of the comment box, and
(d) the cramped line spacing in the comment box.
If anyone can suggest a way to improve any of these items, particularly (a), I shall incorporate it directly into my comment workflow. You could then be assured that even my most logorrheal comments would include grammatical apostrophes rather than catastrophes.
posted by Pastabagel at 11:22 AM on September 29, 2009 [1 favorite]
> Personally I think the claim that removing the 500 character limit on YouTube or similar comments would directly improve comment quality is dubious.
I agree, he's identifying correlation where causation isn't determinable. For example there are community norms, volume of traffic, and topic ranges that also dictate commenter behavior.
But what I get out of it is that putting a low character limit on comments is not a useful means of controlling comment quality. Projecting from that, community norms, at least, are modulated by the participants, and if the participants can't write at length when the subject demands it, they're going to give up trying.
posted by ardgedee at 11:38 AM on September 29, 2009
I agree, he's identifying correlation where causation isn't determinable. For example there are community norms, volume of traffic, and topic ranges that also dictate commenter behavior.
But what I get out of it is that putting a low character limit on comments is not a useful means of controlling comment quality. Projecting from that, community norms, at least, are modulated by the participants, and if the participants can't write at length when the subject demands it, they're going to give up trying.
posted by ardgedee at 11:38 AM on September 29, 2009
(I've written that blog post)
"Personally I think the claim that removing the 500 character limit on YouTube or similar comments would directly improve comment quality is dubious."
I don't make that claim. That's why I specifically say that my recommendation only applies to sites which don't have a malevolent audience. Having said that, YouTube's comments also show a slight correlation between comment length and quality.
I agree, he's identifying correlation where causation isn't determinable.
True, and I don't claim that there is causation. The basic question I was trying to answer is whether it makes sense to restrict comment length, and the data seems to suggest that it doesn't. That's pretty much all I'm claiming.
posted by L_K_M at 12:42 PM on September 29, 2009
"Personally I think the claim that removing the 500 character limit on YouTube or similar comments would directly improve comment quality is dubious."
I don't make that claim. That's why I specifically say that my recommendation only applies to sites which don't have a malevolent audience. Having said that, YouTube's comments also show a slight correlation between comment length and quality.
I agree, he's identifying correlation where causation isn't determinable.
True, and I don't claim that there is causation. The basic question I was trying to answer is whether it makes sense to restrict comment length, and the data seems to suggest that it doesn't. That's pretty much all I'm claiming.
posted by L_K_M at 12:42 PM on September 29, 2009
@Blazecock Pileon: Specifically, how would you improve this experiment?
posted by L_K_M at 12:43 PM on September 29, 2009
posted by L_K_M at 12:43 PM on September 29, 2009
"His data show that long comments are good for MetaFilter, but they also show that length is largely irrelevant to YouTube. Presumably 5000-character YouTube comments would be as inane as 20-character comments."
The data shows a slight correlation between length and quality even on YouTube. However, I do point out that my recommendation only applies if you expect to have a reasonably positive community of commenters. I agree that removing the length restriction on YouTube would probably not result in more coherent comments, but merely in longer incoherent comments.
posted by L_K_M at 12:49 PM on September 29, 2009
The data shows a slight correlation between length and quality even on YouTube. However, I do point out that my recommendation only applies if you expect to have a reasonably positive community of commenters. I agree that removing the length restriction on YouTube would probably not result in more coherent comments, but merely in longer incoherent comments.
posted by L_K_M at 12:49 PM on September 29, 2009
The basic question I was trying to answer is whether it makes sense to restrict comment length, and the data seems to suggest that it doesn't. That's pretty much all I'm claiming.
I don't really see how the data backs that up though, other than that MetaFilter comments that would be over the limit on other sites are rated highly. From the conclusion:
Since longer comments on average were rated higher, restricting comment length may cut off the best comments, or force comment writers to redact their comments until they fit – and it’s pretty hard to trim a 2000-character-text into 500 characters without removing substance.
I actually think this is probably true, but since the study did not test for this it's not really proven by the actual data. It's possible that limiting the maximum length of each entry would result in tighter writing, or more engagement due to the threads being easier to read, or some other positive effect that would make the overall comment quality better. This particular dataset does not cover cases where the comment quality is high enough to take advantage of longer comment lengths but where technical length limitations exist, so any conclusions about the worth of such limits or recommended limits is conjecture.
Also, I think the method of rating these comments by displaying them completely out of context of the thread/video heavily favors longer comments (rather than being a "slight disadvantage" as presented in the post), so I'm not sure if the data that was collected is a true measure of comment quality.
posted by burnmp3s at 1:03 PM on September 29, 2009
I don't really see how the data backs that up though, other than that MetaFilter comments that would be over the limit on other sites are rated highly. From the conclusion:
Since longer comments on average were rated higher, restricting comment length may cut off the best comments, or force comment writers to redact their comments until they fit – and it’s pretty hard to trim a 2000-character-text into 500 characters without removing substance.
I actually think this is probably true, but since the study did not test for this it's not really proven by the actual data. It's possible that limiting the maximum length of each entry would result in tighter writing, or more engagement due to the threads being easier to read, or some other positive effect that would make the overall comment quality better. This particular dataset does not cover cases where the comment quality is high enough to take advantage of longer comment lengths but where technical length limitations exist, so any conclusions about the worth of such limits or recommended limits is conjecture.
Also, I think the method of rating these comments by displaying them completely out of context of the thread/video heavily favors longer comments (rather than being a "slight disadvantage" as presented in the post), so I'm not sure if the data that was collected is a true measure of comment quality.
posted by burnmp3s at 1:03 PM on September 29, 2009
"it's not really proven by the actual data."
This is true, the data doesn't prove anything, and I don't claim it does. But I do think it suggests that restricting comment length is misguided. The data isn't proof for anything, but I think it is an indication of what's going on.
"It's possible that limiting the maximum length of each entry would result in tighter writing, or more engagement due to the threads being easier to read, or some other positive effect that would make the overall comment quality better."
Yes, this is possible. I can't think of any examples of websites that seem to exhibit this effect, but it's possible. Any ideas how this could be tested?
"This particular dataset does not cover cases where the comment quality is high enough to take advantage of longer comment lengths but where technical length limitations exist, so any conclusions about the worth of such limits or recommended limits is conjecture."
Yes, it's conjecture, but it's informed conjecture :-)
How would you set this up in a way that would account for the problems you point out?
"Also, I think the method of rating these comments by displaying them completely out of context of the thread/video heavily favors longer comments (rather than being a "slight disadvantage" as presented in the post), so I'm not sure if the data that was collected is a true measure of comment quality."
Yes, there were some factors which could have advantaged or disadvantaged longer comments. I'm not sure how to account for the lack of context. Any ideas?
posted by L_K_M at 1:14 PM on September 29, 2009
This is true, the data doesn't prove anything, and I don't claim it does. But I do think it suggests that restricting comment length is misguided. The data isn't proof for anything, but I think it is an indication of what's going on.
"It's possible that limiting the maximum length of each entry would result in tighter writing, or more engagement due to the threads being easier to read, or some other positive effect that would make the overall comment quality better."
Yes, this is possible. I can't think of any examples of websites that seem to exhibit this effect, but it's possible. Any ideas how this could be tested?
"This particular dataset does not cover cases where the comment quality is high enough to take advantage of longer comment lengths but where technical length limitations exist, so any conclusions about the worth of such limits or recommended limits is conjecture."
Yes, it's conjecture, but it's informed conjecture :-)
How would you set this up in a way that would account for the problems you point out?
"Also, I think the method of rating these comments by displaying them completely out of context of the thread/video heavily favors longer comments (rather than being a "slight disadvantage" as presented in the post), so I'm not sure if the data that was collected is a true measure of comment quality."
Yes, there were some factors which could have advantaged or disadvantaged longer comments. I'm not sure how to account for the lack of context. Any ideas?
posted by L_K_M at 1:14 PM on September 29, 2009
This is true, the data doesn't prove anything, and I don't claim it does. But I do think it suggests that restricting comment length is misguided. The data isn't proof for anything, but I think it is an indication of what's going on.
Yeah, I think I was actually being more critical of this because it gives the impression of "I have this opinion about technical comment length limitations, and here is the proof" when the graphs and data that collected aren't much more compelling than just showing some really great 1000+ character comments and making the argument that they wouldn't be possible with a limit. Of course having at least some data to objectively show that long comments can be some of the best comments on certain sites is better than nothing, but when I see graphs and statistics used to back up a claim I have a tendency to look at the data skeptically, even if I think the claim is true.
Any ideas how this could be tested?
Not easily. The best opportunity for studying the effect of the limitations would be to look at a site before and after restrictions were imposed/lifted.
I'm not sure how to account for the lack of context. Any ideas?
I rated a few of the comments when this was posted to MetaTalk before, but I gave up because it was too hard to figure out what the comments were even talking about, let alone rate them. Presenting the user with the post/video and the properly formatted comment would go a long way towards recreating the context. A lot of comments on both MetaFilter and YouTube would make no sense at all without at least knowing what the topic is. Really though, it's very difficult to make any kind of qualification about comments without reading the whole thread, or at least the parts of the thread that are relevant to any given comment. Showing some of the comments around the comment being rated would probably be a good start. It would also be easier to do on sites where the comments are threaded.
posted by burnmp3s at 1:50 PM on September 29, 2009
Yeah, I think I was actually being more critical of this because it gives the impression of "I have this opinion about technical comment length limitations, and here is the proof" when the graphs and data that collected aren't much more compelling than just showing some really great 1000+ character comments and making the argument that they wouldn't be possible with a limit. Of course having at least some data to objectively show that long comments can be some of the best comments on certain sites is better than nothing, but when I see graphs and statistics used to back up a claim I have a tendency to look at the data skeptically, even if I think the claim is true.
Any ideas how this could be tested?
Not easily. The best opportunity for studying the effect of the limitations would be to look at a site before and after restrictions were imposed/lifted.
I'm not sure how to account for the lack of context. Any ideas?
I rated a few of the comments when this was posted to MetaTalk before, but I gave up because it was too hard to figure out what the comments were even talking about, let alone rate them. Presenting the user with the post/video and the properly formatted comment would go a long way towards recreating the context. A lot of comments on both MetaFilter and YouTube would make no sense at all without at least knowing what the topic is. Really though, it's very difficult to make any kind of qualification about comments without reading the whole thread, or at least the parts of the thread that are relevant to any given comment. Showing some of the comments around the comment being rated would probably be a good start. It would also be easier to do on sites where the comments are threaded.
posted by burnmp3s at 1:50 PM on September 29, 2009
"The best opportunity for studying the effect of the limitations would be to look at a site before and after restrictions were imposed/lifted."
Sounds like a long-term experiment :-)
"Presenting the user with the post/video and the properly formatted comment would go a long way towards recreating the context. (…) Showing some of the comments around the comment being rated would probably be a good start."
Yes, giving the original post or video the comment relates to, or surrounding comments, is a good idea and should be possible. And yes, I definitely screwed up when I removed the HTML from the comments. I didn't want MeFi comments to look differently from YouTube comments, but in hindsight, that was a pretty dumb decision.
"Really though, it's very difficult to make any kind of qualification about comments without reading the whole thread, or at least the parts of the thread that are relevant to any given comment."
So it would make sense to reuse ratings made in the context of the comment on the site the comment originally appeared on. Perhaps using a Slashdot-like site would work, where people already rate comments. That's definitely feasible.
That way, one could also objectively test whether changing comment restriction changes average comment ratings. That would be a pretty huge project for a pretty small result, though.
posted by L_K_M at 2:05 PM on September 29, 2009
Sounds like a long-term experiment :-)
"Presenting the user with the post/video and the properly formatted comment would go a long way towards recreating the context. (…) Showing some of the comments around the comment being rated would probably be a good start."
Yes, giving the original post or video the comment relates to, or surrounding comments, is a good idea and should be possible. And yes, I definitely screwed up when I removed the HTML from the comments. I didn't want MeFi comments to look differently from YouTube comments, but in hindsight, that was a pretty dumb decision.
"Really though, it's very difficult to make any kind of qualification about comments without reading the whole thread, or at least the parts of the thread that are relevant to any given comment."
So it would make sense to reuse ratings made in the context of the comment on the site the comment originally appeared on. Perhaps using a Slashdot-like site would work, where people already rate comments. That's definitely feasible.
That way, one could also objectively test whether changing comment restriction changes average comment ratings. That would be a pretty huge project for a pretty small result, though.
posted by L_K_M at 2:05 PM on September 29, 2009
Sounds like a long-term experiment :-)
If you knew a certain site that changed its comment length limits on a specific date, and the comments had timestamps, you could pretty easily scrape comments from before and after the change to compare them.
posted by burnmp3s at 2:30 PM on September 29, 2009
If you knew a certain site that changed its comment length limits on a specific date, and the comments had timestamps, you could pretty easily scrape comments from before and after the change to compare them.
posted by burnmp3s at 2:30 PM on September 29, 2009
If you knew a certain site that changed its comment length limits on a specific date, and the comments had timestamps, you could pretty easily scrape comments from before and after the change to compare them.
True. Any sites that come to mind? I don't think I can think of any site that introduced, changed or removed comment length limits, though I'm sure there are many which did...
posted by L_K_M at 2:32 PM on September 29, 2009
True. Any sites that come to mind? I don't think I can think of any site that introduced, changed or removed comment length limits, though I'm sure there are many which did...
posted by L_K_M at 2:32 PM on September 29, 2009
Specifically, how would you improve this experiment?
People already take for granted as common knowledge that YouTube comments are of worse subjective quality than comments from practically any other online forum, including Metafilter. You do not appear to take into account this "background" knowledge people bring to the table when they participate in your experiment.
More specifically, comments that are written as if they were posted on YouTube would, I expect, get voted with a lower quality score, regardless of their length. I would imagine that, for example, comparing:
"Neo ain't got shit on this guy :p"
with an information-less comment like:
"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
I'll bet that the first style of comment would still get a lower aggregate score.
That's just one source of a priori bias on the part of voters that I think would need to be tested as an effect (using random or Markov-generated comments, for example, as controls) before you could begin to evaluate comment length as a significant effect.
There are other effects. Metafilter comments originate with a very specific demographic of commenters, who are, all else the same, probably older and more educated, and who, as a result, are likely more comfortable with expressing complex ideas without regard for length within the flow of a larger conversation.
When a short comment is pulled at random — and what is defined as random comment sampling? — does its length ignore its potentially critical, i.e. "high quality" contribution to a larger thread of discussion? It would seem hard to evaluate this effect without having voters evaluate whole threads.
Independent of length, comments may not be hermetic, gut-level expressions of a single thought, the way they are, on the whole, on YouTube. The "big picture" nature of how site members contribute within a thread can be a more intangible and difficult effect to suss out.
When the commenters come to the sites to contribute, do the educational or other experiential backgrounds result in comments of different quality levels, up front? Perhaps mixing Metafilter comments with those from other sites with contributors with similar backgrounds would establish the independence of this effect on comment length.
Is there something about the culture at Metafilter that promotes long comments of a certain quality level, compared with the body of commenters who participate on other non-YouTube sites? Does your hypothesis test the quality of long comments at Metafilter, or long comments on web forums in general? Doing these sorts of controls will help you focus your hypothesis.
I think you already pointed out a couple testing issues, such as your self-selected pool of voters (vote early, vote often!) which could cause a bias on results.
When doing a correlation test, you can also test the significance level of your results, to determine if you could have obtained your correlation result from random chance. It may be a matter of the luck of the draw that you obtained the results that you did. A correlation coefficient of 0.4 might look significant to you, but without a p-value may in fact have been obtained in a fairly wide region of insignificance.
posted by Blazecock Pileon at 3:04 PM on September 29, 2009 [2 favorites]
People already take for granted as common knowledge that YouTube comments are of worse subjective quality than comments from practically any other online forum, including Metafilter. You do not appear to take into account this "background" knowledge people bring to the table when they participate in your experiment.
More specifically, comments that are written as if they were posted on YouTube would, I expect, get voted with a lower quality score, regardless of their length. I would imagine that, for example, comparing:
"Neo ain't got shit on this guy :p"
with an information-less comment like:
"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
I'll bet that the first style of comment would still get a lower aggregate score.
That's just one source of a priori bias on the part of voters that I think would need to be tested as an effect (using random or Markov-generated comments, for example, as controls) before you could begin to evaluate comment length as a significant effect.
There are other effects. Metafilter comments originate with a very specific demographic of commenters, who are, all else the same, probably older and more educated, and who, as a result, are likely more comfortable with expressing complex ideas without regard for length within the flow of a larger conversation.
When a short comment is pulled at random — and what is defined as random comment sampling? — does its length ignore its potentially critical, i.e. "high quality" contribution to a larger thread of discussion? It would seem hard to evaluate this effect without having voters evaluate whole threads.
Independent of length, comments may not be hermetic, gut-level expressions of a single thought, the way they are, on the whole, on YouTube. The "big picture" nature of how site members contribute within a thread can be a more intangible and difficult effect to suss out.
When the commenters come to the sites to contribute, do the educational or other experiential backgrounds result in comments of different quality levels, up front? Perhaps mixing Metafilter comments with those from other sites with contributors with similar backgrounds would establish the independence of this effect on comment length.
Is there something about the culture at Metafilter that promotes long comments of a certain quality level, compared with the body of commenters who participate on other non-YouTube sites? Does your hypothesis test the quality of long comments at Metafilter, or long comments on web forums in general? Doing these sorts of controls will help you focus your hypothesis.
I think you already pointed out a couple testing issues, such as your self-selected pool of voters (vote early, vote often!) which could cause a bias on results.
When doing a correlation test, you can also test the significance level of your results, to determine if you could have obtained your correlation result from random chance. It may be a matter of the luck of the draw that you obtained the results that you did. A correlation coefficient of 0.4 might look significant to you, but without a p-value may in fact have been obtained in a fairly wide region of insignificance.
posted by Blazecock Pileon at 3:04 PM on September 29, 2009 [2 favorites]
"I'll bet that the first style of comment would still get a lower aggregate score."
It is probably true that length is one of the ways people decided how to rate comments. Given the same content, a longer comment might have gotten a more positive rating based on its length alone. However, if people think that longer comments are better merely because they are longer, that actually supports my conclusion that we should not force commenters to keep comments short.
"Metafilter comments originate with a very specific demographic of commenters, who are, all else the same, probably older and more educated, and who, as a result, are likely more comfortable with expressing complex ideas without regard for length within the flow of a larger conversation."
Yes. It's true that the comments I used may not be representative of comments on other sites, or even of comments on MetaFilter a year ago, or in a year.
"When a short comment is pulled at random — and what is defined as random comment sampling? — does its length ignore its potentially critical, i.e. "high quality" contribution to a larger thread of discussion?"
This is true, comments lacked context, and this probably affected shorter comments more than longer comments. For a future experiment, it would be better to look at comments that were rated in the context of their thread, for example on Slashdot.
So, all things considered, when redoing the experiment, I should:
- Use comments from a wide variety of sites
- Use ratings the actual users of the site made in the original context of the comment
Do you think that would give valid results?
posted by L_K_M at 3:24 PM on September 29, 2009
It is probably true that length is one of the ways people decided how to rate comments. Given the same content, a longer comment might have gotten a more positive rating based on its length alone. However, if people think that longer comments are better merely because they are longer, that actually supports my conclusion that we should not force commenters to keep comments short.
"Metafilter comments originate with a very specific demographic of commenters, who are, all else the same, probably older and more educated, and who, as a result, are likely more comfortable with expressing complex ideas without regard for length within the flow of a larger conversation."
Yes. It's true that the comments I used may not be representative of comments on other sites, or even of comments on MetaFilter a year ago, or in a year.
"When a short comment is pulled at random — and what is defined as random comment sampling? — does its length ignore its potentially critical, i.e. "high quality" contribution to a larger thread of discussion?"
This is true, comments lacked context, and this probably affected shorter comments more than longer comments. For a future experiment, it would be better to look at comments that were rated in the context of their thread, for example on Slashdot.
So, all things considered, when redoing the experiment, I should:
- Use comments from a wide variety of sites
- Use ratings the actual users of the site made in the original context of the comment
Do you think that would give valid results?
posted by L_K_M at 3:24 PM on September 29, 2009
If you knew a certain site that changed its comment length limits on a specific date, and the comments had timestamps, you could pretty easily scrape comments from before and after the change to compare them.
True. Any sites that come to mind? I don't think I can think of any site that introduced, changed or removed comment length limits, though I'm sure there are many which did...
If I recall correctly, The A.V. Club started limiting comment length immediately following this article, where some prick posted the entirety of the book Pinnochio. I don't know what the limit is now, unfortunately, but The A.V. Club is also, I think, a better point of comparison to Metafilter because the general tone and quality of the comments runs much, much closer than, say, youtube.
As it is, the data collected here is certainly interesting, but the overall conclusions, whether stated explicitly or merely "suggested" comes off a little bit like "after studying the body-mass index of regular patrons of Hardee's and Paul's Vegan Cafe, I can suggest that smaller portions are healthier." While that may or may not be true, the bigger issue seems to be that not eating at Hardee's is probably healthier.
Still, as I said, this is all interesting, but I think the methodology needs some tweaking.
posted by Navelgazer at 3:31 PM on September 29, 2009
True. Any sites that come to mind? I don't think I can think of any site that introduced, changed or removed comment length limits, though I'm sure there are many which did...
If I recall correctly, The A.V. Club started limiting comment length immediately following this article, where some prick posted the entirety of the book Pinnochio. I don't know what the limit is now, unfortunately, but The A.V. Club is also, I think, a better point of comparison to Metafilter because the general tone and quality of the comments runs much, much closer than, say, youtube.
As it is, the data collected here is certainly interesting, but the overall conclusions, whether stated explicitly or merely "suggested" comes off a little bit like "after studying the body-mass index of regular patrons of Hardee's and Paul's Vegan Cafe, I can suggest that smaller portions are healthier." While that may or may not be true, the bigger issue seems to be that not eating at Hardee's is probably healthier.
Still, as I said, this is all interesting, but I think the methodology needs some tweaking.
posted by Navelgazer at 3:31 PM on September 29, 2009
Comparisons made in the linked document are meaningless without presentation of measures of statistical significance. Until appropriate statistical testing is performed, I rate this analysis Moderate Frowny Face.
posted by monocyte at 3:38 PM on September 29, 2009 [1 favorite]
posted by monocyte at 3:38 PM on September 29, 2009 [1 favorite]
As it is, the data collected here is certainly interesting, but the overall conclusions, whether stated explicitly or merely "suggested" comes off a little bit like "after studying the body-mass index of regular patrons of Hardee's and Paul's Vegan Cafe, I can suggest that smaller portions are healthier."
I can't disagree with that. It's certainly true that the correlation between perceived quality and length does not occur on all sites (or does not occur on all sites to the same extent), as the YouTube data shows.
posted by L_K_M at 3:39 PM on September 29, 2009
I can't disagree with that. It's certainly true that the correlation between perceived quality and length does not occur on all sites (or does not occur on all sites to the same extent), as the YouTube data shows.
posted by L_K_M at 3:39 PM on September 29, 2009
The biggest problem, though, seems to be that longer comments will provide their own context, whereas shorter comments will rely on those comments which came before, as a general correlation. When we are asked to rate comments free of any context (and I did some the other day when I first saw this) that difference can be huge in determining how highly we rate something. I wish I had a god idea of how to solve this.
posted by Navelgazer at 3:45 PM on September 29, 2009 [1 favorite]
posted by Navelgazer at 3:45 PM on September 29, 2009 [1 favorite]
However, the «ratings growth» flattens out at a comment length of about 2000 characters
tl;dr
posted by Pinback at 4:26 PM on September 29, 2009
tl;dr
posted by Pinback at 4:26 PM on September 29, 2009
However, if people think that longer comments are better merely because they are longer, that actually supports my conclusion that we should not force commenters to keep comments short.
Designing your experiment procedure so as to prove your conclusion is not the correct thing to do.
I think you need to think carefully about your controls, to account for voter bias and from bias due to the source and context of the messages changing the score independent of comment length. Getting this right is critical to successfully testing your hypothesis.
You will also want to randomly select comments across multiple sites, and you will want to randomly select voters.
Ultimately, you want to procedurally isolate the contribution that length-in-itself makes to subjective-comment-scoring, as much as possible, before you begin to calculate correlation.
Once you have a properly designed experiment and have calculated correlation, you can then calculate the statistical significance of your experiment results, to determine whether or not your hypothesis can be supported by the evidence you have collected.
posted by Blazecock Pileon at 4:26 PM on September 29, 2009
Designing your experiment procedure so as to prove your conclusion is not the correct thing to do.
I think you need to think carefully about your controls, to account for voter bias and from bias due to the source and context of the messages changing the score independent of comment length. Getting this right is critical to successfully testing your hypothesis.
You will also want to randomly select comments across multiple sites, and you will want to randomly select voters.
Ultimately, you want to procedurally isolate the contribution that length-in-itself makes to subjective-comment-scoring, as much as possible, before you begin to calculate correlation.
Once you have a properly designed experiment and have calculated correlation, you can then calculate the statistical significance of your experiment results, to determine whether or not your hypothesis can be supported by the evidence you have collected.
posted by Blazecock Pileon at 4:26 PM on September 29, 2009
The comments my ass makes when visiting the little boy's room are both shorter and better than typical YouTube comments.
The same applies to any give "news outlet" comment section.
posted by edgeways at 5:06 PM on September 29, 2009
The same applies to any give "news outlet" comment section.
posted by edgeways at 5:06 PM on September 29, 2009
The comments my ass makes when visiting the little boy's room are both shorter and better than typical YouTube comments.
You might want to get that looked at, it could be the sign of a more serious condition.
posted by juv3nal at 5:35 PM on September 29, 2009
You might want to get that looked at, it could be the sign of a more serious condition.
posted by juv3nal at 5:35 PM on September 29, 2009
I’ve looked at four different punctuation mistakes: A missing apostrophe in «don’t», an ellipsis with only two dots, an ellipsis with four dots, and four consecutive question marks (arguably, the last one isn’t really a mistake, I guess).I wonder if he's counting an ellipsis at the end of a sentence, and thus followed by a period, as "an ellipsis with four dots"....
posted by Flunkie at 6:17 PM on September 29, 2009
Does it make sense that the comment length on an exponential scale follows a normal distribution? Or is that weird?
posted by smackfu at 6:29 PM on September 29, 2009
posted by smackfu at 6:29 PM on September 29, 2009
The word douchebag improves comment quality. Love it.
posted by jessamyn (staff) at 7:26 PM on September 29, 2009
posted by jessamyn (staff) at 7:26 PM on September 29, 2009
hey L_K_M, thanks for doing this - it's fun. it's just the sort of thing I would wonder about, but never get around to actually doing.
posted by 5_13_23_42_69_666 at 7:28 PM on September 29, 2009
posted by 5_13_23_42_69_666 at 7:28 PM on September 29, 2009
Offhand I would say that it is the $5 buy-in and wanting to maintain your reputation as well as Moderation and the threat of banning rather than a character limit that leads to comments that are more thoughtful and hence lengthier and ranked higher. It would be a trivial matter to write a thoughtful (long) comment and post it in successive 500 character posts.
I would be interested to see a rerun of the data limited to the time period before September 26, 2009 at 5:57 PM as I douchebaggedly rated the comments.
posted by vapidave at 10:10 PM on September 29, 2009
I would be interested to see a rerun of the data limited to the time period before September 26, 2009 at 5:57 PM as I douchebaggedly rated the comments.
posted by vapidave at 10:10 PM on September 29, 2009
I find the rate-the-comment icons ... disturbing.
posted by potsmokinghippieoverlord at 10:26 PM on September 29, 2009
posted by potsmokinghippieoverlord at 10:26 PM on September 29, 2009
@Blazecock Pileon:
I said: "However, if people think that longer comments are better merely because they are longer, that actually supports my conclusion that we should not force commenters to keep comments short."
You said: "Designing your experiment procedure so as to prove your conclusion is not the correct thing to do."
I acknowledge that there are huge issues with the test, but I most certainly didn't design it in a way that I expected would influence the results. I actually believe that the ratings are reasonably good despite of the problems like lack of context, since they correlate well with other properties I would expect to be a predictor for quality, like sentence length and spelling mistakes. Of course, it could be that people rated comments based on such properties because the comments lacked context.
At any rate, I want to make it perfectly clear that I did not intend to influence the results by my setup. This isn't a scientific study, it's a small weekend project. Most of the problems with the setup are a direct result of the fact that the time between coming up with the idea and starting to implement it was perhaps half a day. I simply didn't spend enough time thinking about it.
I ran this test because I actually wanted to know how the data would look – not because I wanted to prove my point. I did release the YouTube data, which shows practically no correlation between length and quality. I did not put as much thought into it as I should have, and I made a few obvious mistakes (like stripping HTML from comments, which probably hurt longer comments because they are more likely to contain quotes that look strange without HTML), but I did not intend to influence the results.
If I come across as defensive on this, it's because I think it's a slightly unfair criticism, and I feel kind of strongly about it. I'm grateful for all the criticism of the setup itself, you guys made a lot of great points, and if I repeat the test, it'll be much better. But I want to make it perfectly clear that I did not intentionally screw up the setup.
"Ultimately, you want to procedurally isolate the contribution that length-in-itself makes to subjective-comment-scoring, as much as possible, before you begin to calculate correlation."
Do you think something like adding lorem ipsum comments of random length to the corpus would work for that? If people rated nonsensical short comments worse than nonsensical long comments, that bias would become obvious.
"You will also want to randomly select comments across multiple sites, and you will want to randomly select voters."
A more random selection of comments should be feasible. I'm not sure how to randomly select voters on a project like this one. Any ideas?
posted by L_K_M at 11:46 PM on September 29, 2009
I said: "However, if people think that longer comments are better merely because they are longer, that actually supports my conclusion that we should not force commenters to keep comments short."
You said: "Designing your experiment procedure so as to prove your conclusion is not the correct thing to do."
I acknowledge that there are huge issues with the test, but I most certainly didn't design it in a way that I expected would influence the results. I actually believe that the ratings are reasonably good despite of the problems like lack of context, since they correlate well with other properties I would expect to be a predictor for quality, like sentence length and spelling mistakes. Of course, it could be that people rated comments based on such properties because the comments lacked context.
At any rate, I want to make it perfectly clear that I did not intend to influence the results by my setup. This isn't a scientific study, it's a small weekend project. Most of the problems with the setup are a direct result of the fact that the time between coming up with the idea and starting to implement it was perhaps half a day. I simply didn't spend enough time thinking about it.
I ran this test because I actually wanted to know how the data would look – not because I wanted to prove my point. I did release the YouTube data, which shows practically no correlation between length and quality. I did not put as much thought into it as I should have, and I made a few obvious mistakes (like stripping HTML from comments, which probably hurt longer comments because they are more likely to contain quotes that look strange without HTML), but I did not intend to influence the results.
If I come across as defensive on this, it's because I think it's a slightly unfair criticism, and I feel kind of strongly about it. I'm grateful for all the criticism of the setup itself, you guys made a lot of great points, and if I repeat the test, it'll be much better. But I want to make it perfectly clear that I did not intentionally screw up the setup.
"Ultimately, you want to procedurally isolate the contribution that length-in-itself makes to subjective-comment-scoring, as much as possible, before you begin to calculate correlation."
Do you think something like adding lorem ipsum comments of random length to the corpus would work for that? If people rated nonsensical short comments worse than nonsensical long comments, that bias would become obvious.
"You will also want to randomly select comments across multiple sites, and you will want to randomly select voters."
A more random selection of comments should be feasible. I'm not sure how to randomly select voters on a project like this one. Any ideas?
posted by L_K_M at 11:46 PM on September 29, 2009
@Flunkie: "I wonder if he's counting an ellipsis at the end of a sentence, and thus followed by a period, as "an ellipsis with four dots"...."
Everything with four or more consecutive dots was included in that group. Admittedly, this does falsely include this valid use of ellipses and counts it as a mistake, as long as the writer used three dots instead of an ellpisis character "…".
@vapidave: "I would be interested to see a rerun of the data limited to the time period before September 26, 2009 at 5:57 PM as I douchebaggedly rated the comments."
Unfortunately, I started the poll on this day, so there was no data before September 26. At any rate, I have now deleted the data, so I can't do further tests on it.
@potsmokinghippieoverlord: I apologize :-)
@5_13_23_42_69_666: Thank you!
I'll probably be away from keyboard for about a day, so if you have further questions, I apologize in advance for not replying sooner. Thanks for all the comments and criticism!
posted by L_K_M at 11:55 PM on September 29, 2009
Everything with four or more consecutive dots was included in that group. Admittedly, this does falsely include this valid use of ellipses and counts it as a mistake, as long as the writer used three dots instead of an ellpisis character "…".
@vapidave: "I would be interested to see a rerun of the data limited to the time period before September 26, 2009 at 5:57 PM as I douchebaggedly rated the comments."
Unfortunately, I started the poll on this day, so there was no data before September 26. At any rate, I have now deleted the data, so I can't do further tests on it.
@potsmokinghippieoverlord: I apologize :-)
@5_13_23_42_69_666: Thank you!
I'll probably be away from keyboard for about a day, so if you have further questions, I apologize in advance for not replying sooner. Thanks for all the comments and criticism!
posted by L_K_M at 11:55 PM on September 29, 2009
You are not logged in, either login or create an account to post comments
posted by fightorflight at 9:13 AM on September 29, 2009