Notes from the Metafilter research panel at AoIR's 12th conference

November 18, 2011 3:55 PM

So, four mefites collaborated on a Metafilter-themed panel last month at at the Association of Internet Researchers's 12th annual Internet Research conference, and we thought we'd go ahead and share our presentation notes and such. Come on in!

The three actual academics involved are iamkimiam (Kim), lewistate (Quinn), and DiscourseMarker (Kris); they were nice enough to invite me to show up and talk about the Infodump as well. The four of us took turns talking for about fifteen minutes to a few dozen fellow nerds including some other mefites, and it was a fun time.

We've each got presentation notes available, as well as audio from the actual presentation. The presentations, in the order they were given:

- Josh: The Metafilter Infodump
- Kim: Why the way you say MeFi matters
- Kris: Anonymity and Conflict in Metafilter.com
- Quinn: The Rhetoric of Trust

Audio from the panel (very lofi, there was a seriously noisy HVAC fan running on and off):

- Kris's intro
- Josh's talk
- Kim's talk
- Kris' talk
- Quinn's talk
- Q & A
- Kim's rerecorded talk

Some of this research you may remember hearing about previously: Quinn's work originally came up in Metatalk in October, 2009 when he was conducing a survey for his dissertation research (which subsequently got him his Ph.D); last March we helped Kim run a poll about "mefi" and "mefite" pronunciations for her sociolinguistics research toward her Masters. I'm not sure where/when on the site Kris has spoken specifically about this part of her work but she's totally a nerdy doctor too, so, hey.

posted by cortex to MetaFilter-Related at 3:55 PM (59 comments total)

Oh, excellent - I've been looking forward to hearing more about this panel.
posted by rtha at 3:58 PM on November 18, 2011

Cheers for that, will have a look at this later - there's quite a lot of room for research on various aspects of mefi, never mind the whole internet.
posted by sgt.serenity at 4:19 PM on November 18, 2011

Neat and sweet! I ♥ research. Computer, send to this to the computer and print it out.
posted by Toekneesan at 4:46 PM on November 18, 2011 [1 favorite]

Yay!
posted by nangar at 4:57 PM on November 18, 2011

Ha. We are the M-Set!

I demand a t-shirt.
posted by DarlingBri at 5:00 PM on November 18, 2011 [2 favorites]

I am distressingly interested in all these things.
posted by The Whelk at 5:04 PM on November 18, 2011 [1 favorite]

I would like to receive your newsletter.
posted by Rumple at 5:04 PM on November 18, 2011

We are MeFi! Occupy MeFi!

Hey, it rhymes!

(well, at least half of it does.)
posted by iamkimiam at 5:06 PM on November 18, 2011 [1 favorite]

Informative and interesting. Thanks.
posted by waviolet at 5:06 PM on November 18, 2011

NERDS! I dig it profusely.
posted by zomg at 5:10 PM on November 18, 2011

"We are MeFi! Occupy MeFi!

Hey, it rhymes!"

It does! Established by SCIENCE
posted by Blasdelb at 5:12 PM on November 18, 2011

I would so wear an M-Set t-shirt.
posted by rtha at 5:15 PM on November 18, 2011

How do you view Kris's slide show? It just toggle back and forth between the first and second slide when I tap them.
posted by unliteral at 5:25 PM on November 18, 2011

The little arrows-in-circles buttons are working for me to move forward and backward a slide at time.
posted by cortex (staff) at 5:30 PM on November 18, 2011

Can we get a nicely linked list of the available dissertations from these academics? Because I am done with all four of these interesting presentations and would like more please.
posted by DarlingBri at 5:33 PM on November 18, 2011

From cortex's presentation notes:

One of our members, Plutor, put together a nice graph that uses the benchmark that a user is “active” if and only if they’ve had some sort of site activity (posting, commenting, favoring) in the last month [...]

Heh, I guess great minds think alike, because that graph looks exactly like the one I made.
posted by FishBike at 5:46 PM on November 18, 2011 [1 favorite]

ALL YOU NERDS LOOK THE SAME TO ME and I hadn't gotten much sleep the day I was writing that up and dammit, baby, I'm sorry, I never meant to hurt you.

Fixed.
posted by cortex (staff) at 5:50 PM on November 18, 2011

I don't think the data showing that more engaged and least anonymous members were the most aggressive in the favorites thread is all that surprising in context. The more engaged you are with the site, the more importance you are likely to assign to policy and structural issues and the more you consider yourself to have a personal stake in the outcome. The more personal your stake in an issue the more likely you are to have a strong emotional reaction. The stronger your emotional reaction to something, the more likely you are to display aggression if that reaction is negative.
posted by Justinian at 5:54 PM on November 18, 2011

There's only one way to say MeFi.
posted by cjorgensen at 6:22 PM on November 18, 2011

Thanks for pulling all of this stuff together, cortex! The AoIR panel was a lot of fun — it almost feels like cheating to list it on my c.v.
posted by lewistate at 6:36 PM on November 18, 2011

Nifty. If anyone's interested in working with the Corpus - which doesn't seem to be listed as one of cortex's projects - you might like the free Natural Language Processing online class that Stanford's offering in January.

I tweeted this at cortex & iamkimiam, with a borked link.
posted by Pronoiac at 6:48 PM on November 18, 2011 [2 favorites]

cjorgensen: MeFi

Yeah, it's pronounced "clickclick click clickclick click."
posted by Kattullus at 6:48 PM on November 18, 2011 [1 favorite]

I pronounce it as a series of timed pheromone bursts.
posted by The Whelk at 6:56 PM on November 18, 2011 [1 favorite]

wow. Manning and Jurafsky wrote the textbooks on natural language processing: Foundations of Statistical Natural Language Processing and Speech and Language Processing.
posted by research monkey at 7:30 PM on November 18, 2011 [1 favorite]

Mnemonic for my personal pronunciation:

To me, me fie. Fie on other pronunciations.
posted by Flunkie at 8:06 PM on November 18, 2011 [1 favorite]

I'm glad everyone is enjoying this stuff (so far)! If you are really interested, I have more stuff on my website. I also now have a full paper for the MeFi study. It's not online, but I would be happy to email anyone who is interested a copy if you MeFiMail me your email address.

I don't think the data showing that more engaged and least anonymous members were the most aggressive in the favorites thread is all that surprising in context

Yes, it does make sense in context for sure. What's interesting, though, is that here on MeFi it seems to work in the opposite direction from what the literature would generally predict, wi regards to anonymity. Most of the research on anonymity points to more anonymity leading to more participation, rather than less. So basically, that leaves use with...

MetaFilter: Refuting the literature since 1999.
posted by DiscourseMarker at 9:13 PM on November 18, 2011 [4 favorites]

Researchers's

what
posted by Sys Rq at 9:41 PM on November 18, 2011

Most of the research on anonymity points to more anonymity leading to more participation, rather than less.

I think more people participating, not people participating more, no? Ie: more anonymity = 100 people posting once, instead of 10 people posting 10 times each.
posted by empath at 9:46 PM on November 18, 2011

This is interesting stuff, thanks guys.
posted by arcticseal at 10:52 PM on November 18, 2011

I just wish I hadn't already been in the workforce by the time this newfangled thing emerged... I wish/woulda/coulda studied all these lovely patterns. Maybe after I'm 50
posted by infini at 11:17 PM on November 18, 2011

If you're not paying attention to Kim's slide titles, you're.. not paying attention well enough. My favorite (so far) is MeTa Data Meta-Data. COMIC GENIUS. Surely more comments to come as I continue to intake caffeine and talks. (This sounds suspiciously like work on a weekend.)
posted by knile at 12:34 AM on November 19, 2011

Yeah, I can confirm that that amazing graph previously attributed to me was not mine. The perfectly straight line indicating that there's no "best Asking time": all me.

I think your talk deserves more attributions, cortex.
posted by Plutor at 5:14 AM on November 19, 2011

The problem with "M-set" as a group name or t-shirt is that it can too easily be vandalized by dropping a mocking "T" in the middle, though some may find that a feature rather than a bug.

Speaking of things one shouldn't do, part of me now can't wait until the next time someone asks when the best Asking time is.
posted by MCMikeNamara at 5:44 AM on November 19, 2011 [1 favorite]

Heh. I never though of that. The "empty set". Or perhaps the M+set. Your bug is my favorite.
posted by iamkimiam at 6:56 AM on November 19, 2011

I think your talk deserves more attributions, cortex.

Yeah, I should go back and mark those up more clearly. The irony is that the blog post is basically a transcript of my edited talk because the one that I wrote up for my first draft was about 50% too long, and like an idiot I overwrote the original draft with the trimmed one and lost a lot of those little details in the process.
posted by cortex (staff) at 7:10 AM on November 19, 2011

I thought this bit from DiscourseMarker in the Q&A was fascinating:

The first idea was ... I did this whole qualitative analysis of the thread and developed all these wonderful categories and we had all these human coders who failed utterly to get any intercoder reliability, like, whatsoever. And that was the first key about context. Because they were just, you know, graduate students. And we trained them. and we trained them again, and then we trained them again, and we had like Skype conversations. And, like, they couldn't see it, because they were just not members of the community, and they couldn't agree on anything.

So, it was also like ... how do you ... So the community has some definition of hostility. How do you get inside that? Right? And how do figure out what it is that this community believes is acceptable behavior ....

So group of grad students who aren't familiar with the community or the commenters can't tell if someone is being hostile or not, even after they've been trained on a coding system.

But we can tell, right? I'm not so sure, really. Knowing the commenters presumably helps some, but I'm not sure we'd get much better inter-rater reliability than Marker's grad students. We're always admonishing each other (or being admonished) to 'assume good faith' and give each other the benefit of the doubt, and accusing people reading other people's comments 'in the worst possible light'. We're constantly having to decide whether a question is hostile or rhetorical, or just a question, whether the factoid somebody brought up in a comment was subtle dig at a point we made earlier, or just a comment. Sometimes we guess wrong.

To me, this underlines how much we miss without cues like intonation, facial expression, eye-contact and gesture. We all start out with as much familiarity with the site as DiscourseMarker's grad-student coders; I'm not sure we actually get much better with practice. In the absence of the cues we rely on to interpret people's tone and intent in face-to-face conversation, our perception of hostility frequently depends on our assumptions and what we anticipate.
posted by nangar at 7:11 AM on November 19, 2011 [2 favorites]

The Further Research questions on iamkimiam's presentation made me wonder if there's been any study of bad-faith participation in online community-building.
posted by carsonb at 7:20 AM on November 19, 2011

- Kris: Anonymity and Conflict in Metafilter.com

OK! I'm just gonna keep reading/listening, heh. This is all incredibly interesting, thanks for putting it together!
posted by carsonb at 7:24 AM on November 19, 2011

Oh, and John Logie, one of the folks responsible for this Q&A site comparison research, dropped by after the talk to nerd out a little about the results from that.
posted by cortex (staff) at 7:24 AM on November 19, 2011 [1 favorite]

That Quintilian quote might make a good posting page flag where we now have the "Everybody needs a hug".

All this species of eloquence, however, requires the speaker to be a man of good character and of pleasing manners. The virtues which he ought to praise, if possible, in his client, he should possess or be thought to possess himself. Thus he will be a great support to the causes that he undertakes, to which he will bring credit by his own excellent qualities.
posted by bukvich at 7:38 AM on November 19, 2011 [3 favorites]

Also if a guy got an academic publication largely based upon the Russian sex traffic incident, did he ever write up documentation that there were actual real Russian sex traffic people involved or is that just too tangled up to determine?
posted by bukvich at 7:45 AM on November 19, 2011

If I recall right, I think Quinn's work didn't really have anything to do with the Russian Girls episode itself; that was just something that happened that tied in well to the whole trust and ethos thing when he was already in the tail end of his dissertation process but made for a more interesting capstone story during our panel than his more research-focused talk, after we'd all sort of gone RESEARCH RESEARCH RESEARCH for fortyfive minutes before he was up.
posted by cortex (staff) at 7:57 AM on November 19, 2011

Also if a guy got an academic publication largely based upon the Russian sex traffic incident ...

He didn't. Although Quin/lewistate talked about the Russian girls incident in his presentation, it was an illustration. His dissertation was on MetaTalk and site ethos. (In other words, what cortex said.) There a MeTa thread about it here, and the pdf is here.
posted by nangar at 8:07 AM on November 19, 2011

I pronounced MeFi like mofo.
posted by 23skidoo at 8:19 AM on November 19, 2011 [1 favorite]

Kim (or whoever): is there a transcript of her talk somewhere? I have problems watching videos of any length.
posted by saveyoursanity at 9:33 AM on November 19, 2011

You know, I don't participate in many online communities, but I get the impression that the kind of self-reflection which MetaFilter engages in (and which is excellently sampled in these talks) is kind of rare overall.

I think it's a great and excellent thing that not only does something like MetaTalk exist for this website, but that people are so actively engaged with trying to examine and understand more deeply this community.

Thanks so much for posting this. I've felt like I'm part of this crowd for a while now, but I feel that much more deeply after going through the material in this post. Such a cool place -- thanks for existing!
posted by hippybear at 9:41 AM on November 19, 2011 [2 favorites]

Me-fie!

/holds out pinky.
posted by babbyʼ); Drop table users; -- at 9:53 AM on November 19, 2011

saveyoursanity, I may transcribe it at some point. There are two audio files linked up there, if you just want to listen to it. The rerecorded one is better, despite my cold. The original is massively annoying, IMHO...I'm going a million miles an hour and littering my speech with ums and uhs. Gotta work on that.
posted by iamkimiam at 10:52 AM on November 19, 2011

The re-recorded one is da bomb, Kim.
posted by Mister_A at 11:31 AM on November 19, 2011

did he ever write up documentation that there were actual real Russian sex traffic people involved or is that just too tangled up to determine?

Just to piggyback on what cortex and nangar said, the Russian sex trafficking incident happened after I stopped formally collecting data for my study, so it isn't actually mentioned in my dissertation. The story was written up in Mother Jones and Newsweek, but even if I hadn't read those articles, I would be convinced that the story was true. I watched the original thread (and the related MetaTalk thread) unfold in realtime—as I was proofreading my dissertation, appropriately enough—and there's no doubt in my mind that those girls genuinely were in danger.

I went to the AoIR conference with two versions of my presentation: one chock-full of data and one that told a great story that illustrated the concept of "collective ethos" better than any chart or graph could. Right before our panel, I asked Kris, Kim, and Josh if they wanted to hear the data version or the story version. I'm glad they picked the story version.
posted by lewistate at 6:00 PM on November 19, 2011

Thanks for this, I've been mulling some of this stuff over on my head since you guys told me about it at the meetup.
posted by P.o.B. at 10:20 PM on November 19, 2011

> there's no doubt in my mind that those girls genuinely were in danger.

I am asking the question because I have doubts and I don't find the evidence I have seen convincing in the slightest. To me the biggest piece of evidence was people were asking questions and getting confusing answers, and the true explanation of that may well be that the Russian natives they were talking to didn't speak very good English. One of the girls spoke very little English at all as I recall.
posted by bukvich at 8:06 AM on November 20, 2011

bukvich, если хотите, мы можем говорить на русском языке. Я хорошо говорю по-русски. And I had no problem communicating with my former student.

With all respect due to your concerns, I ask that you take into consideration that the State Department, the DHS ICE Squad, the NYPD, RCLC (and many other people and organizations who were working with the girls directly) felt that the girls were in enough danger to intervene, and that in every way this was a textbook case of trafficking. Those organizations, in many cases, had more information than I did. The fact that the girls remained unharmed is a testament to the whole system working, badly as it did/does.

In the followups that I posted, and in the later news coverage (linked by Quinn above, but particularly, read this piece), reporters vetted the facts - not just with me, but with outside parties - and presented detail not available in the original thread. The original thread is inevitably incomplete because I posted it all, in real-time, from a phone, and because I had to be cautious about their safety.

I've spent a lot of time here trying to answer to the missing pieces, and to be fair to skeptics, but there will always be some unsatisfied people no matter what I do. To be honest with you, I get the strong sense that some people would not believe it unless the girls were actually prostituted, and escaped or later found dead. And even then, there would be victim-blaming. I'm not in any way saying that is your response -- I am saying that the remarkable thing here is that they WEREN'T hurt, in the sense that they were forced into prostitution, because the effort here kept them from it. As pollomacho (a rep of the State Department) said: "Usually I get these cases weeks or months after they've stepped off that bus.". We got them in the nick of time, thanks to Kathrine.

This morning, I listened to Quinn's discussion about community and his framing and retelling of the incident, and I thought it was very well considered (though it was very stressful for me to relive - somehow, someone else telling the story so well put me right back into it). Great work, Quinn, and thank you for your insights and for framing it all in terms of the strength of this community and online community in general. It bears repeating.

And thanks for sharing the talks here, Cortex.
posted by fake at 3:17 PM on November 20, 2011 [3 favorites]

Thanks for your kind words, fake, and sorry for the stress! I think you're right — this is one of those incidents that perfectly illustrates why MetaFilter is worth taking seriously, not just as a research site, but as a force for good in MeFites' lives (and, in this case, the lives of MeFites' friends). I love all of the datawankery that can be done with the Infodump, but for my money, it's the human stories that make MetaFilter worth studying.
posted by lewistate at 6:07 PM on November 20, 2011

As someone in school who has been reading around 3-4 hours of research daily for the past three months, this is something fun to look forward to over Thanksgiving Break! And yay for stats/data analysis class paying off!
posted by smirkette at 8:10 PM on November 20, 2011

I updated my writeup with captions attributing the various graphs and stuff. So now I'm both (a) correctly identifying FishBike as making the stuff that FishBike made and (b) successfully crediting Plutor with the stuff that Plutor made, which is a better deal all around than lying and saying Plutor made what FishBike made.
posted by cortex (staff) at 9:31 AM on November 21, 2011

[...] which is a better deal all around than lying and saying Plutor made what FishBike made.

... unless there are any errors, in which case Plutor made that one.

Thanks for posting all this here. It's been really fun reading through and listening to these presentations.

This sounds suspiciously like work on a weekend.

Heh, listening to people talk who clearly know what they're talking about is pretty much the opposite of what it's like to be at work, for me. It's quite relaxing!
posted by FishBike at 9:57 AM on November 21, 2011

At last! My favorite MetaFilter stat has finally appeared again! Ever since saw this originally (years ago, I can't remember when) I've always entered the site through MetaTalk, so if I left the site before visiting the blue or green, I could be a part of this ultraminority.

I am the 0.2%!!!
posted by slogger at 1:02 PM on November 21, 2011 [1 favorite]

But, slogger, as oddly addictive as this niche is, what keeps you away from the other good stuff?
posted by sammyo at 8:31 AM on November 24, 2011