Comments on: Highlighted terms in search results

Highlighted terms in search results

stebulus — Sat, 11 Aug 2012 14:35:33 -0800

The site search sometimes highlights words as hits which are not hits.

Examples:

In the results for {fled}, the word "flamboyant" is also highlighted.
In the results for {sled}, the words "slipped", "slot", and "slaughter" are also highlighted.
In the results for {cred}, the words "create", "CraigsList", "creepy", etc. are also highlighted.
In the results for {bling}, the words "black", "blop", and "blouse" are also highlighted.
In the results for {ching}, the words "challenge", "Chatterjee", "Christine", etc. are also highlighted.
In the results for {wooing}, the words "wooden" and "WooTube" are also highlighted.

Similarly in {fred}, {fling}, {sting}, {aping}. User's uninformed speculation as to the cause: (I understand that no bug report is complete without this.) I suppose the code sees the terminal "ed" of "fled" (for example), figures that "fled" is the past tense form of a verb "fl", then helpfully tries to highlight alternatively inflected forms, which it identifies by pattern "fl*" (in glob syntax). Search results themselves are fine: The posts returned by the search do all actually seem to contain the searched-for words — for example, the results for {fled} have "flamboyant" highlighted, but they don't include posts that have the word "flamboyant" but not the word "fled". Further evidence that this is just about the highlighting and not the search proper is the results for {natural thing}, in which even stopwords such as "the" and "their" are highlighted. (I add the word "natural" to the search only to reduce the number of hits below the threshold where the search will refuse to return results.) Counterexamples showing some limits of the phenomenon: It doesn't happen when there's just one letter before the presumed suffix, e.g., in the results for {med}, not all words starting with "m" are highlighted (though the word "meds" is), and in the results for {bing}, not all words starting with "b" are highlighted. In the results for {tees}, the word "TeeFury" is highlighted, but the word "Texas" is not, suggesting that terminal "s" is treated as a productive suffix but terminal "es" is not.

By: pb

pb — Sat, 11 Aug 2012 14:38:01 -0800

Thanks for the heads up. We'll get this fixed up.

By: xorry

xorry — Sat, 11 Aug 2012 15:05:07 -0800

I think the term for this is 'stemming' it has never occured to use it for highlighting
Cool.

By: stebulus

stebulus — Sat, 11 Aug 2012 15:08:57 -0800

Right, "stemming". I'd forgotten that word.

Added it as a tag. Thanks.

By: LobsterMitten

LobsterMitten — Sat, 11 Aug 2012 15:36:23 -0800

stebulus, I enjoy your explanation of this problem.

By: pb

pb — Sat, 11 Aug 2012 16:16:14 -0800

ok, I updated the way we're highlighting search results. It should look a little better, but let me know if you spot anything off.

Yeah, we were using my own stemmer for highlighting search results. It wasn't that great compared to something that's been around forever like the Porter Stemming algorithm. But that's out of service now so let's see how this new method works.

By: pb

pb — Sat, 11 Aug 2012 16:18:39 -0800

Oh, and by "out of service" I mean my code. The Porter Stemming algorithm is still the way to go.

By: stebulus

stebulus — Sat, 11 Aug 2012 16:41:53 -0800

It should look a little better, but let me know if you spot anything off.

Yup, all the false positives I reported seem to be gone.

Another matter which I noticed while gathering evidence about the stemming thing: this post contains the words "fled" and "flamboyant"; it shows up in a search for {fled}, and in a search for {flamboyant}, but strangely, not in a search for {fled flamboyant}.

By: pb

pb — Sat, 11 Aug 2012 16:48:57 -0800

huh, that is odd. I'm not sure why that's happening offhand. I'll see if I can track it down.

By: OmieWise

OmieWise — Sat, 11 Aug 2012 18:10:26 -0800

That was some fast work on a Saturday night.

By: MattMangels

MattMangels — Sat, 11 Aug 2012 20:33:02 -0800

MetaFilter's search function is awful, just awful. It really ought to be replaced with a google site search, which is what I end up using anyway since the native search function can't find shit.

By: cjorgensen

cjorgensen — Sat, 11 Aug 2012 20:38:23 -0800

Some of us hate google too!

By: cortex

cortex — Sat, 11 Aug 2012 21:02:16 -0800

MetaFilter's search function is awful, just awful. It really ought to be replaced with a google site search, which is what I end up using anyway since the native search function can't find shit.

For the record, the internal search function actually functions pretty rock solid for what it does; google site search, as featured as it is in terms of query flexibility (and very useful in particular for quoted phrases) is a shitshow in terms of complete coverage and is incapable of providing results parsed at a per-comment level. It, for many specific search purposes, is the awful tool.

Fortunately, both exist and careful searches can use each for the situations where it excels.

By: infini

infini — Sat, 11 Aug 2012 23:57:36 -0800

Google's search is much worse and the highlighting of random words unconnected to your search term does not have a helpful pb in the background fixing it. Thank god for verbatim.

By: a humble nudibranch

a humble nudibranch — Sun, 12 Aug 2012 02:19:55 -0800

god had nothing to do with it

By: maxwelton

maxwelton — Sun, 12 Aug 2012 04:20:28 -0800

Your god had nothing to do with it. I was saved when I accepted pb into my heart (also my: hearts, hearted, hearten, heartize, hearting, and pb's acres o' heart).

By: item

item — Sun, 12 Aug 2012 07:30:25 -0800

The site search must be broken, because my name doesn't come up when you enter 'awesome', 'amazing', or 'downright righteous mofo' into the query box. I assume this is some type of coding error and I eagerly await its correction.

By: infini

infini — Sun, 12 Aug 2012 08:03:21 -0800

*shoves other reading glasses back on bridge of nose*

*peers down item list*

*scrolls further*

By: benito.strauss

benito.strauss — Sun, 12 Aug 2012 09:07:40 -0800

I would just like to say that "Porter Stemming" sounds like the name of a 1930s Yale literary critic.

By: stebulus

stebulus — Sun, 12 Aug 2012 09:19:22 -0800

Fortunately, both exist and careful searches can use each for the situations where it excels.

I wonder if you could expand on this a bit. I know it can get pretty frustrating when you can't figure out how to get a search function to answer the question you want to ask, and often search functions are poorly documented, so it's hard to improve your search experience by learning more. Maybe this is an opportunity for us to understand the available tools better.

What does the site search do, exactly, and when is it better than google?

By: languagehat

languagehat — Sun, 12 Aug 2012 09:34:32 -0800

You know what wasn't a hit that should have been a hit? "Sheena Is a Punk Rocker." That still pisses me off.

By: pb

pb — Sun, 12 Aug 2012 09:36:58 -0800

The main difference between the site search and Google is that Google indexes pages, the site search indexes atomic elements like posts and comments. So the search results here will point precisely to a post or comment. The search results at Google could point to any page on the site.

So if you do a search at Google for "Obama", it's going to cast a wide net. You will get any page that mentions the word "Obama". That means the word could appear in comments, posts, archive pages, tag pages, etc. When you click the result, you'll need to scan the page to find that occurrence of the word.

If you do a search for "Obama" in the local search, you can choose between getting posts with the word "Obama" or comments with the word "Obama". Each search result points directly to that discrete bit of text. We order the results by date posted, so if you have a sense of the timeframe you're interested in, that can be better. Google uses its pagerank algorithm to sort results, and there isn't much awareness of when something was posted. Also, we know we're searching the entire database with every local search. Google has scraped the pages from the outside, and could be missing pieces. It's generally very good, but it has its own rules about what is indexed and what isn't. We don't have any rules for inclusion and we don't have to discover things—everything is indexed.

As cortex said, sometimes you might want a wide net, popular items, and more flexibility in how you search. Sometimes you might have a sense of what you're after and get better results locally.

By: iamkimiam

iamkimiam — Sun, 12 Aug 2012 09:47:20 -0800

Text world problems.

By: iamkimiam

iamkimiam — Sun, 12 Aug 2012 09:48:07 -0800

Damn. That would have been better as "First word problems."

By: stebulus

stebulus — Sun, 12 Aug 2012 09:57:29 -0800

Thanks, pb.

By: stebulus

stebulus — Sun, 12 Aug 2012 10:01:49 -0800

You know what wasn't a hit that should have been a hit? "Sheena Is a Punk Rocker." That still pisses me off.

Once upon a time, Tony Dillon Davis of CKUA, after playing some song (I don't recall what song), said, "What I find most baffling about that song is that it was not a hit."

By: jessamyn

jessamyn — Sun, 12 Aug 2012 10:17:26 -0800

That would have been better as "First word problems."

While I am somewhat tired of the idea of first world problems, I like the idea of text world problems and think they encompass a lot of my personal woes, so I will take this up as my personal whinge, even though it may not have been an intended phrase coinage.

By: infini

infini — Sun, 12 Aug 2012 10:32:26 -0800

*iamkimiam scribbles notes hastily into margin of dissertation draft*

By: trip and a half

trip and a half — Sun, 12 Aug 2012 11:09:15 -0800

I have thoroughly enjoyed this thread.

By: benito.strauss

benito.strauss — Sun, 12 Aug 2012 11:09:45 -0800

Actually, "First word problem" is something I suffer from regularly. "Well,", "So,", "But,", "Also,", even "Actually,". The main use I see for the edit window is deleting them after I post.

By: stebulus

stebulus — Sun, 12 Aug 2012 11:22:31 -0800

The novel of a thousand pages begins with a single word.

By: iamkimiam

iamkimiam — Sun, 12 Aug 2012 11:23:53 -0800

I actually stopped the mad scribbling to fascinate on the word "whinge". So strange, so high-scoring in Scrabble.

Oh yeah, I verbed that.

By: iamkimiam

iamkimiam — Sun, 12 Aug 2012 11:24:18 -0800

So does the one of six.

By: iamkimiam

iamkimiam — Sun, 12 Aug 2012 11:25:30 -0800

Crap. You said pages, not words. I'll quit while I'm not dead last on the comprehension skills.

By: stebulus

stebulus — Sun, 12 Aug 2012 11:25:43 -0800

One of Six was my favourite Borg.

By: iamkimiam

iamkimiam — Sun, 12 Aug 2012 11:29:46 -0800

First Contact problems.

By: stebulus

stebulus — Sun, 12 Aug 2012 11:30:41 -0800

Oh yeah, I verbed that.

It has a nice flavour from the influence of "fixate", actually. Verb away.

(I mean, like, "go ahead and verb", not like, "get thee behind me, verb!", nor like, "to the verb cave!")

By: iamkimiam

iamkimiam — Sun, 12 Aug 2012 11:34:31 -0800

I initially interpreted as a cross between "bombs away!" and "go away!", which both involve running, but for completely different reasons.

By: stebulus

stebulus — Sun, 12 Aug 2012 11:53:30 -0800

I was going to bow out of the thread, but I have a Last Word problem.

By: infini

infini — Sun, 12 Aug 2012 12:31:31 -0800

Mine is third verb problem

By: cgc373

cgc373 — Sun, 12 Aug 2012 15:09:32 -0800

Minus fourth surd wobble.

By: OmieWise

OmieWise — Sun, 12 Aug 2012 15:13:15 -0800

I'm having a beer because I've got a thirsty nerd problem.

By: iamkimiam

iamkimiam — Sun, 12 Aug 2012 15:19:13 -0800

I'll have a Bond-style martini, because I've got an overused cliché problem.

By: item

item — Sun, 12 Aug 2012 21:02:24 -0800

I just drank a glass of Metamucil because I have a hard turd problem.

By: cortex

cortex — Sun, 12 Aug 2012 21:47:54 -0800

I'm trying to break into songwriting for Catholic missals and I've got Bob Hurd problems.

By: benito.strauss

benito.strauss — Sun, 12 Aug 2012 22:03:59 -0800

I'm having the most trouble putting away this flag. I've got the worst furled problem.

By: EmpressCallipygos

EmpressCallipygos — Sun, 12 Aug 2012 22:05:42 -0800

Someone asked me to make a sculpture of an eagle out of sausage and I'm not sure how. I've got a wurst bird problem.

By: cortex

cortex — Sun, 12 Aug 2012 22:10:56 -0800

A client asked me to make them a paisley mink, so I got fur swirled problems.

By: benito.strauss

benito.strauss — Sun, 12 Aug 2012 22:41:32 -0800

Hey, who threw my Auden book in the blender? Now I've got a verse whirled problem.

By: infini

infini — Mon, 13 Aug 2012 02:39:58 -0800

Your book went flying, right out the window, now its a worse wold problem.

By: stebulus

stebulus — Mon, 13 Aug 2012 08:26:55 -0800

I got scared by some dancing Ents, but I guess it's just a firs twirled problem.

By: infini

infini — Mon, 13 Aug 2012 08:44:12 -0800

You win, stebulus.

By: cavalier

cavalier — Mon, 13 Aug 2012 10:22:12 -0800

Oh, you guys. <3

By: iamkimiam

iamkimiam — Mon, 13 Aug 2012 12:02:44 -0800

While they were out dancing, I was singing with Treebird about his old birch killed problem.

By: pb

pb — Mon, 13 Aug 2012 15:14:20 -0800

The fled flamboyant problem should be fixed.

By: stebulus

stebulus — Mon, 13 Aug 2012 15:51:05 -0800

The fled flamboyant problem should be fixed.

Looks to be fixed, indeed. Cool.

Just out of curiosity, what was going on there?

By: pb

pb — Mon, 13 Aug 2012 15:56:52 -0800

Just out of curiosity, what was going on there?

The terms were in separate columns and the AND query was running against each column separately. So the hit would only come back if the terms both appeared in any one column. I needed to reorganize things a bit so the search engine treats the components of the post as one piece of text.

By: stebulus

stebulus — Mon, 13 Aug 2012 16:00:09 -0800

Aha. Neat.

Thanks.