Idea: Metafilter Link Checker (Needs a Fancier Name)

September 18, 2009 8:59 PM

Idea: the Metafilter link-checker. Take the part of the 'new post' process that double-checks against prior posts, add the part of Mefi that conjures up the related posts, put a link to the feature up on the two lines of links near the logo.

Maybe add auto-links to site-specific searches via Google and Yahoo just for convenience's sake.

Yes, there are other ways to do it, including prepping a new post. But it'd make it a few degrees easier: as it stands now, in order to get to the part of the Mefi posting process that double-checks your link against doubles, you essentially have to write it up first, and THEN get it rejected for doublehood.

This'd be a much easier way to quickly ascertain that something's a double before writing up a good post.

And if the "related post" stuff was added to it, hopefully whatever algorithm's in use there might be able to give a clue as to whether the subject matter was handled before, even if the link itself wasn't a double.

posted by WCityMike to feature requests at 8:59 PM (42 comments total)

Not speaking to the widget idea at all, but I find that I can pretty much find anything by doing a tag search for the two or three most relevant tags for my post topic and Google gets the rest. And, sometimes I cheat and just put all my links in the box and press preview and see if they've been posted before.
posted by jessamyn at 9:03 PM on September 18

Yeah, there's definitely existing workarounds: I do the second one of your strategies myself a lot.

But this'd have real beneficial use, methinks, and more importantly, I imagine it would be less effort – i.e., just involving splicing together a few already existing features together on one page, as opposed to other pony requests that ask for new things.
posted by WCityMike at 9:09 PM on September 18

I'm not sure how the feature would differ from the existing site search. You can type www.example.com into the current search and get a list of posts that have that link. How would a link-checker be different from that?
posted by pb at 9:20 PM on September 18

Shit. There goes my dissertation on the history of www.example.com in popular culture.
posted by yhbc at 9:31 PM on September 18 [1 favorite]

The way I kind of envision it is something like this textual mockup:

----

Metafilter Link Checker

URL to Check: [http://www.example.com____________________________________]

[OK] [CANCEL]

----

So you click on the URL, and then this page comes up:

---

Link Results

[Copy: These results are meant as an aid and aren't a guarantee it won't be a duplicate blah blah blah blah.]

[The whole "Link Search" section that comes up in the "Step 2" part of the FPP posting process]

[Copy: Don't take the above as a total all-clear -- be sure to double-check yourself using:]

* [Generated link to Metafilter site search for example.com]
* [Generated link to Google search for 'site:metafilter.com link:example.com'] (or "site:ask.metafilter.com", etc.)
* [Generated link to Yahoo search for 'site:metafilter.com link:example.com'] (or "site:ask.metafilter.com", etc.)

[Copy: These posts may have covered the same subject matter (i.e., a video on a different host, etc.):]

['Related posts' section -- maybe run on the text from the permalink URL -- i.e. http://www.example.com/2009/08/12/this-is-an-example-permalink would run the 'Related posts' algorithm on words "example this is an example permalink" -- or if that's too hassleworthy, then you could abandon this part of the idea, since there wouldn't be written description text yet, if that's what the 'related posts' algorithm uses]

----

I don't know, I just think this'd be useful because it'd be one process by which you could collect all the strategies together, and make them noticeable.

And by doing that small amount of work, I just have a hunch it might make a impact on the number of dupe posts you guys have to delete.

I don't know the stats, but it seems like even those have been around a while can oft fall victim to the dupe goblin (I almost did a dupe tonight -- a good 10-15 minutes after I started this MeTa post, not before, it didn't prompt this idea -- with that "What's in the Box?" CGI video that was back in April ... because MeFi's link was a YouTube post, and so it didn't trigger the site's dupe checker -- but in that case, the 'Google' search would've caught me).
posted by WCityMike at 9:42 PM on September 18

I suppose I see an upside for admins and an upside for Mefites.

The upside for Mefites would be that you could really quickly check a link and get a fairly definitive answer and/or easy way to use most of the dupe-checking tools at your disposal pretty quickly, without having to go through the "writeup" stage of a FPP.

The upside for admins is that hopefully this ease would lead to a reduction in dupes, which seem to compromise a fair amount of the post deletions necessary (at least from an informal glance at the posts using the Metafilter Deleted Posts Greasemonkey script).

The second upside for admins is that I don't see this as requiring a ton of new stuff to implement: it strikes my (admittedly layman's) eyes as something that would be mostly just a splicing job of existing features.
posted by WCityMike at 9:44 PM on September 18

hopefully this ease would lead to a reduction in dupes

It can't possibly lead to a reduction in dupes if it's using the same algorithm that is already is in effect when making a post. All posts already go through the existing dupe checker logic, so if it's a dupe which would have not been detected (e.g. non-canonical domain names, different query params, same-content-different-site) then it's not going to be magically detected by putting it through that same exact logic twice.
posted by Rhomboid at 9:53 PM on September 18

I see that it'd frame the search results in terms of looking for links, but I don't think the effect would be much different than the current search results. And even at the bottom of every current search results page is, "You can also try a Google search for www.example.com limited to MetaFilter" that links to the Google search. (Sticking with the example.com theme we have going.)

And I'm not sure that breaking this out from the posting process would help. You can run a quick link or two through the standard search before you start writing a post, and then you'll have the benefit of a search that checks every link in your post as you're submitting your post. If people won't use the standard search or check similar tags now, I don't think they would use yet another specialized search that just checks for duplicate links when they're going to get that as they post anyway.

We do get duplicate posts that waste a bit of time and I think it's good to think about ways to reduce them, but I think we have a number of options in place already.
posted by pb at 9:54 PM on September 18

I'm against anything that lowers the bar for effort needs expending when making a post to MeFi. This coming from a poster who expends the least amount of effort necessary to get something on the blue. Also, site search is well past the point of being difficult to use. As long as you're logged in, you don't even have to input the www. or the .wevs—just put the meat of the URL (eg. kernsholler) in the search box and keep an eye out for '[keyword in HTML]'. If it's harder for people who want to post links to frequently-linked sites, all the better.
How am I so socially and financially liberal, yet so MetaFilterally conservative?
posted by carsonb at 9:58 PM on September 18

And, sometimes I cheat and just put all my links in the box and press preview and see if they've been posted before.

Bingo. Anytime I want to make a post, I do the above first, before even trying to coherent a write sentence.

I'm not sure what problem your solution would fix Mike.
posted by Brandon Blatcher at 10:11 PM on September 18

Fine, let's put the idea aside, then.

I must admit that this thread leaves me pretty considerably frustrated. I'm not sure if it's frustration aimed at the admins or the community – and please note I mean this to be frustration expressed in a courteous way, not a flameout insultfest. I don't mean anything that follows to be a personal attack, or personally insulting, and hopefully none of it comes across that way.

I respect that Metafilter has a tradition behind it, and a simplicity to guard -- but I think, like any positive attribute, taking that to too far of an extreme can turn a virtue into a vice.

This idea here strikes me as one of the relatively more simple ideas proposed. To me, it strikes me as both low effort and high benefit – and few ideas are around that have that high of a benefit-to-effort ratio. And the fact that even it ...

Metafilter's strength is not in its UI but in the quality of its moderation, and the quality and variety of its users and the content that such quality and variety produces.
But I do think that there's something a little wrong with the state of affairs (and I do mean "little" in a literal, not hyperbolic, way) when Plutor's Greasemonkey scripts are responsible for the more useful updates to Metafilter's user functionality.

Cortex loves to crunch numbers, and I know he's on vacation now and not likely to see this thread or to expend the effort to do the crunching. But I must admit, I would honestly be very curious to know two statistics about the "feature requests" subcategory of Metafilter. The statistics would be: (a) how many of them have, within say 1-5 posts of its start, a unfavorable comment from an admin suggesting the feature won't be implemented, and (b) how many of the features suggested by users actually saw their way into the Metafilter UI. I tend to think that the statistic for (a) would be easily above 95% and more likely near 99%, and the statistic for (b) would be near zero.

And, fine, if that's the way that the state of affairs on Metafilter is, then that is the way it is. The opinion of the many outweighs the opinion of the few, or the one, and I've got utterly no problem with that equation. But if that's the state of affairs here, then that's the way it should be represented by both admin and community.

I've been in two other environments where companies put forward the untrue concept that they were interested in and wanted others' ideas for improved functionality of their product. In both cases (Bare Bones Software's Mailsmith product and Remember the Milk's Ideas forums), both entities have by now an utterly proven track record, despite their corporate line, for ignoring their users' input, many of which contained some really great ideas. And those communities needn't have bothered, because their brainstorms and feedback didn't amount to anything.

Much like all those designers who obviously put an utterly massive amount of work into the redesign contest a few years ago ... as far as I can tell, the site's interface didn't change at all as a result of those designers' free efforts. I don't even want to think about how I'd feel if I were one of those designers.

Please note that there's a really, really solid difference here between what it may seem like I'm saying and what I am saying. I'm not saying that rigorously sticking to a design and feature set is bad. I'm saying that having that sort of design rigidity and yet maintaining a fiction that ideas are welcome and given full consideration is bad. Because, even if that's what you guys will go on to say, I just don't think that's actually the case.

I can't really think of even one MeTa thread where a user proposed an idea that would permanently affect the site's functionality (as opposed to, say, temporary modifications such as the "Cortex Across America" sidebar, etc.), and it was lauded as a cool idea and then implemented (but I'm sure one will now be found to prove me wrong).

( ... now, to wrap things up, let me feed you guys a straight line so this thread doesn't get too heavy ... )

... or, you know, i could just perpetually have sucky ideas ...
posted by WCityMike at 10:39 PM on September 18 [1 favorite]

This idea kinda took off.
posted by tellurian at 10:51 PM on September 18 [1 favorite]

Dude, I looked at this and was like, well, I'm against it because you should just do your search anyway, you lazy bastard. And then I thought, well, it's kinda dumb to be against things just because they'd be more convenient, like somehow new members and posters wouldn't be working as hard at making their FPPs and it's effort that counts, dammit.

Then I was like, hey, it's OK as an idea, but y'know, if it were left up to me, we'd never get it because I don't know how to code this shit, and if I did, I'd imagine that I'd have, like, a million more things on my plate before I got to it. So if I really wanted it, I'd teach myself how to code it before getting all insistent and entitled, or more likely just say, Fuck it.
posted by klangklangston at 11:37 PM on September 18

In most software projects, the default answer to feature requests is and should be 'no'.
posted by empath at 11:39 PM on September 18 [4 favorites]

The core of Mike's idea might be good i.e. the part about having an explicit way for a user to search for duplicate links.

As to the rest of your long winded thoughts about the site and whether it accepts user input, I found to be so full of half baked, navel gazing assumptions it instantly snapped me awake this morning, so thank you for that.
posted by Brandon Blatcher at 5:30 AM on September 19

This idea kinda took off too. (One MeTa post, one pony request, one successful implementation - batting 1.000). So kids if you're reading at home, remember - whether you're asking for an extra piece of cake, begging to stay up to watch a movie or requesting a new feature on metafilter, always put sugar atop your pretty please.
posted by cashman at 7:07 AM on September 19

I would honestly be very curious to know two statistics about the "feature requests" subcategory of Metafilter. The statistics would be: (a) how many of them have, within say 1-5 posts of its start, a unfavorable comment from an admin suggesting the feature won't be implemented

You act as though not implementing stupid requests is a problem. If the default answer was yes, we'd have favorites removed this week, and then put back next week.

Then we'd have a digg-style ranking algorithm for posts, then customizable profiles that make even myspace want to puke.

Next we'd have a full gmail interface recreated in mefi mail, which only 17 people would use, but would require 3 months of developer time.

Next we'd be able to sort threads by upvotes which would result in most people missing the whole conversation and chiming in with more asinine comments than ever.

Look you've got a site here that has one full time developer (pb) and probably another half-time developer in mathowie. That's not a lot of coding to go around. I don't think you have any idea how difficult it is to implement features, test them exhaustively, troubleshoot, and fix the inevitable bugs, all on top of the everyday day-to-day stuff that goes into keeping a website running smoothly. They've got a list of plans for the site, and they're plugging away at it, but no, they won't jump at your every whim.

Your five bucks entitles you to contribute to the site, not to have the site developers change things around according to your whims. Deal with it.
posted by chrisamiller at 7:20 AM on September 19

Man, I'm getting tired of whiny complainers.
posted by languagehat at 7:56 AM on September 19

i could just perpetually have sucky ideas ...

Actually, you just have a lot of ideas. And MeFi is a site that doesn't implement a lot of new features regularly. That's totally fine but realistically we implement a small subset of feature requests which means that if you put in a lot of feature requests very few of them will be implemented. This has always been true.

maintaining a fiction that ideas are welcome and given full consideration is bad. Because, even if that's what you guys will go on to say, I just don't think that's actually the case.

The site's only programmer responded to you with interest within 25 minutes of you making this request past midnight on a Friday.

Look WCityMike, I know you're frustrated with how the site responds to you, and I know this is true for other MeTa threads you post. We run a really lean site here and one of our mods is on vacation for the entire month. Every new thing that we roll out requires implementation, testing, FAQ-writing, and then the inevitable debugging and "I liked this site better before..." stuff. It's not just a yes/no "will this feature make the site better/worse?" assessment.

If the community is clamoring for a feature, we often try to make that work. If a few individual users want a feature that other people don't see the benefit of, we're less excited about it. The info dump is public, you're more than welcome to run the MeTa threads through some assessment to see how we roll vis a vis feature implementation, but MeTa is really, as we see it, for suggesting features to the community to see what's what and if people are all like "hey yeah!" then we move forward.

It's not that you don't have a good idea or that we haven't thought about it [in this case, as well as in other cases] but that getting site-wide buy in is a tall hurdle, and one that's infrequently met.
posted by jessamyn at 8:03 AM on September 19

MeTa is really, as we see it, for suggesting features to the community to see what's what and if people are all like "hey yeah!" then we move forward.

We were all like "hell no!" to the removal of the colour shifting feature, but you people didn't give a shit. Did you? You don't care about us. You only care about yourselves.
posted by gman at 8:37 AM on September 19

Jessamyn always had a way of explaining things so I could understand them...

WCityMike, I think the short explanation here seems to be that it ain't really broke, so why fix it?
posted by runningwithscissors at 8:42 AM on September 19

Jessamyn always had a way of explaining things so I could understand them...

Yeah, what happened to that?

Just kidding, the mods here are exceptionally well-spoken. I always look forward to their responses to just about anything.
posted by owtytrof at 9:52 AM on September 19

You know, it might not feel as frustrating if you didn't make so many feature requests. Not to be an asshole or anything. Again. Just saying.
posted by katillathehun at 9:58 AM on September 19

*cuts open WCityMike's skull, pokes around in his ideameats*
posted by Eideteker at 10:35 AM on September 19

*bids goodbye to eide, forever*
posted by special-k at 11:27 AM on September 19

Why you all up in his cortex?!
posted by Brandon Blatcher at 11:29 AM on September 19

How many times have I told you not to drink WCityMike's koolaid?
posted by special-k at 11:43 AM on September 19

special-k: "How many times have I told you not to drink WCityMike's koolaid?"

But it tastes so nice and almondy!
posted by WCityMike at 1:06 PM on September 19

jessamyn: Look WCityMike, I know you're frustrated with how the site responds to you, and I know this is true for other MeTa threads you post.

In many ways the site is the source of some of the best advice, feedback, and information of my life; I can say with no qualms I'm a better person because of this site, and I hope that at times my efforts have had a reciprocal effect in some people's lives.

It'd be more accurate to say that I can at times be frustrated with how various things happen to proceed on the site. I imagine many people experience frustration over their own important pet peeves or causes.

The idea that the community really has to clamor behind a feature in order for it to go forward is an idea that makes sense.

For what it's worth, although I tried to put forth my disagreement with courtesy, I seem to have failed at that, judging from your and others' reaction to my comments.

For that, I am sorry.
posted by WCityMike at 7:03 PM on September 19

Look WCityMike, I probably came off as a little harsh there, but in my defense, so did you. I'm sorry, you're sorry - let's call it a wash and get back to snark-as-usual.
posted by chrisamiller at 3:25 PM on September 20

One of the things that this conversation pointed out to me is how fundamentally different search on MeFi is from a Google search of the site. When pb said "You can type www.example.com into the current search and get a list of posts that have that link," I was actually really surprised. Sure enough, a site search for the main link of my most recent post pops that post right up, where, if I do a Google search, I just get the search engine equivalent of a blank stare.

Because I (like many people, I imagine) am so used to using Google (and other search engines that work similarly), it never occurred to me that MeFi search might work this way. I think one of the reasons why WCityMike's proposal seems like a good idea at first blush (at least to me) is that the unique aspects of MeFi search aren't made explicit. You have to be really paying attention to notice the grey "(keyword in HTML)" notation after a search result to realize what you're getting. Is there some way to make it clear how search works here, so people will know that they essentially have a "link checker" at the bottom of every page, and will understand better which searches work better on the site as opposed to Google?
posted by ocherdraco at 9:30 PM on September 20

I use the the New Post page as WCity Mike's link checker from time to time. I find something I think is interesting and wonder what MeFi had to say about it, so I go to New Post, plug in the link and voilà -- there's the MeFi post about my link. It works fine.
posted by Methylviolet at 11:16 PM on September 20

Good point, ocherdraco. MeFi Search is second nature to me, but we don't even have a FAQ entry about it. Our current FAQ about double posts suggests using the New Post form to search, but the site search would be much easier. (I'm guessing that was written before we had a local site search.) So updating that entry and adding a new one with a more detailed description of the site search could help.
posted by pb at 9:39 AM on September 21

I can totally do that and make it make sense and explain how our search really works.
posted by jessamyn at 9:55 AM on September 21

Excellent. Thanks, guys.
posted by ocherdraco at 10:17 AM on September 21

Yeah, I've dropped an occasional comment explaining this or that difference between google and site search—they each work well for specific tasks, less well for others—and codifying some of that in an easily-accessible place seems like a good plan.

To sort of reply (and reiterate) late to a couple specific points WCityMike was making in argument for the original proposal here:

1. We delete a fair number of doubles, but we don't spend much time deleting them, because they tend to be very straightforward. Maybe 1% of the doubles we delete require more than a glance at the flag queue and in the thread for a "previously" link. So further reducing the double load isn't a big priority; the time savings would be pretty minimal.

2. The doubles we do delete mostly wouldn't show up in a link checker that does what the posting form does now; they tend to be doubles-in-content at different urls. We've thought before about ways to make the existing link-checking a little more robust in its dealings with urls that can be similar but not identical (e.g. the various cruft at the end of a youtube link), but while that might get done at some point it only deals with a very specific subset of these, uh, heteroduples.

3. Generally speaking, "it'd be easy, you just need to combine some code right" isn't a good way to go in arguing for implementation. It might be a snap to do initial implementation, it might be a lot less of a snap, but in any case asserting that it just won't be hard to implement ergo we should implement it is making some pretty big assumptions about the amount of work that might be involved in planning and coding and testing and documenting something.
posted by cortex at 9:43 AM on September 22

I don't think this idea is worth its own MeTa thread*, and since it's tangentially related to this, I'm asking it here: in addition to checking a post for previously posted links, would it be possible for MeFi post preview to tell us if we've duplicated links within our posts? I'm sure that sometimes it's done on purpose, but when I'm building a post (particularly one dealing with a lot of similar URLs, like YouTube links, etc.) it would be nice if the page alerted me that I'd done it.

(I'd also love it if preview would alert me to malformed links—the kind that when you click on them in a post they just lead you back to the post page—which occasionally happen when previewing if a tag was left open or there were other html errors.)

*and besides, someone called me "the pony queen" at the last meetup, so I'm hesitant to reinforce that idea
posted by ocherdraco at 11:17 AM on October 12

Short of implementing a full-blown HTML validator, there isn't a good way to let you know when you have broken HTML. We're hoping that you check out links in preview before you post, and we're happy to clean up HTML problems that slip through on this end.

And I think double links within a post are rare enough that we don't need to check specifically for that. We can definitely remove a link here or there after posting if it happens.
posted by pb at 11:26 AM on October 12

Cool, I'll file that one away as unnecessary. I do check things in preview; I think my penchant for ~25 link posts is more to blame for my borky links than anything else.
posted by ocherdraco at 11:57 AM on October 12

Yeah. Edge case. That's what we keep boxcutters around for; no need for a hydraulic Link De-Duper 3000.
posted by cortex at 1:08 PM on October 12

a hydraulic Link De-Duper 3000 (cortexAdmin)

I am totally asking for one of those for Christmas, along with an Official Red Ryder Carbine-Action Two-Hundred-Shot Range Model Air Rifle.
posted by ocherdraco at 1:36 PM on October 12

You'll dupe your eye out.
posted by cortex at 4:07 PM on October 12 [1 favorite]