Are profiles crawled? February 22, 2011 12:33 PM Subscribe
Are MeFi profiles crawled by search engines? I remember reading that they weren't, but I just googled my real name and found my MeFi profile.
Huh. Weird. Yeah, I thought I remembered that profiles were unindexed as well. Apparently I was wrong?
posted by dersins at 12:38 PM on February 22, 2011
posted by dersins at 12:38 PM on February 22, 2011
We've seen some really weird index-but-only-very-slightly behavior in some cases in the past, where Google will seem to know that a profile page exists for some users but doesn't actually apparently have the page cached or the content of the page indexed. As a result, the profile page "existed" as far as Google search is concerned but didn't have any identifying content associated with it.
It shouldn't happen at all—we explicitly tell search engines not to look there—and I know Matt or pb has tried to figure out why even those weird outlier cases were happening but I don't know if we ever got an answer.
If your actual profile page is showing up with stuff like real name content, that's a different thing than we'd seen previously. If you can send us a little more info via email (assuming you don't want to do it in this thread to be archived for all time) that may help.
posted by cortex (staff) at 12:38 PM on February 22, 2011
It shouldn't happen at all—we explicitly tell search engines not to look there—and I know Matt or pb has tried to figure out why even those weird outlier cases were happening but I don't know if we ever got an answer.
If your actual profile page is showing up with stuff like real name content, that's a different thing than we'd seen previously. If you can send us a little more info via email (assuming you don't want to do it in this thread to be archived for all time) that may help.
posted by cortex (staff) at 12:38 PM on February 22, 2011
Also, it's possible that this is actually just the result of google doing customized search results based on browsing history, though I don't know how that'd work exactly or how we'd be able to conclusively establish that is or isn't happening.
I've verified that searching "site:metafilter.com" and my name brings up my profile, and with your name brings up yours, which is weird and shouldn't be happening. Seems like the same old "not caching, not listing a summary" thing in action, but still.
posted by cortex (staff) at 12:42 PM on February 22, 2011
I've verified that searching "site:metafilter.com" and my name brings up my profile, and with your name brings up yours, which is weird and shouldn't be happening. Seems like the same old "not caching, not listing a summary" thing in action, but still.
posted by cortex (staff) at 12:42 PM on February 22, 2011
cortex, if you google his real name (as listed in his profile) thusly: "real name" site:metafilter.com, it comes up with a single result: his profile.
posted by dersins at 12:44 PM on February 22, 2011
posted by dersins at 12:44 PM on February 22, 2011
Also, just fyi, the first google result for a search of my username is my metafilter profile. Not that I care, but still.
posted by dersins at 12:46 PM on February 22, 2011
posted by dersins at 12:46 PM on February 22, 2011
I think I know why it's doing it in my case. I've put a link to my profile on sites that are indexed in the past. This isn't the case anymore.
They must keep the URL in its database or something like that?
posted by defenestration at 12:53 PM on February 22, 2011
They must keep the URL in its database or something like that?
posted by defenestration at 12:53 PM on February 22, 2011
Does this happen if someone links directly to a MeFi user's profile page from some external page? I saw examples of links to defenestration's profile from places like OKCupid, Twitter, favstar.fm. (As another data point, as far as I can tell, my profile, which also contains my real name, is not indexed in Google.)
posted by aught at 1:00 PM on February 22, 2011
posted by aught at 1:00 PM on February 22, 2011
Yeah, I should make it obvious that I don't really care that my profile shows up Google.
I was just seeking clarification.
posted by defenestration at 1:03 PM on February 22, 2011
I was just seeking clarification.
posted by defenestration at 1:03 PM on February 22, 2011
Not to make light of this (I consider internet privacy a very important issue), but if you google my user name, you get "Who is Edward A Womble", and the presumed address of this person.
With a unique username such as mine, you can definitely see which sites scrape bits off metafilter. Exhibit A, Exhibit B.
posted by a womble is an active kind of sloth at 1:06 PM on February 22, 2011
With a unique username such as mine, you can definitely see which sites scrape bits off metafilter. Exhibit A, Exhibit B.
posted by a womble is an active kind of sloth at 1:06 PM on February 22, 2011
I care. I thought profile pages were ungoogleable. When I google my username, I get my page, and I have zero sites linked to MeFi. I'll be taking down my location info now.
posted by bluefly at 1:06 PM on February 22, 2011 [1 favorite]
posted by bluefly at 1:06 PM on February 22, 2011 [1 favorite]
same information that you get if clicking a name link when logged out.
Someone is not playing nice.
posted by edgeways at 1:06 PM on February 22, 2011
Someone is not playing nice.
posted by edgeways at 1:06 PM on February 22, 2011
My username never brought up my profile before now. I don't know if it really matters but it does seem odd.
posted by snsranch at 1:07 PM on February 22, 2011
posted by snsranch at 1:07 PM on February 22, 2011
The price of fame?
posted by Cranberry at 1:08 PM on February 22, 2011 [1 favorite]
posted by Cranberry at 1:08 PM on February 22, 2011 [1 favorite]
Yeah, googling my user name brings up my profile page as well.
posted by shiu mai baby at 1:11 PM on February 22, 2011
posted by shiu mai baby at 1:11 PM on February 22, 2011
mine's all up in the google via my username. But apparently my username is popular for many other people/things so it's on page 2. And none of that other stuff is me. And my real name isn't in my profile. But my picture is. Maybe was???
posted by atomicstone at 1:11 PM on February 22, 2011
posted by atomicstone at 1:11 PM on February 22, 2011
Seems the robots have stopped following the second law of robots[.txt] ...
posted by Babblesort at 1:11 PM on February 22, 2011 [18 favorites]
posted by Babblesort at 1:11 PM on February 22, 2011 [18 favorites]
Just looking at the HTML source, this suggests that y'all are doing it right. And Metafilter's
What's interesting about that search result is that there is no cached version of it listed, yet there are no cache directives on the HTML source. I do not think it is some other associative mechanism, either, like say a Google Profile link to a MetaFilter URL. I don't have anything linked to my profile but sure enough Teh Goog has seen it.
This is really weird.
Check it, so if I search for
I have seen this before when searching for someone using a
Yeah. There's some kind of pointer out there.
posted by adipocere at 1:12 PM on February 22, 2011
robots.txt file is dandy.What's interesting about that search result is that there is no cached version of it listed, yet there are no cache directives on the HTML source. I do not think it is some other associative mechanism, either, like say a Google Profile link to a MetaFilter URL. I don't have anything linked to my profile but sure enough Teh Goog has seen it.
This is really weird.
Check it, so if I search for
$DEFENESTRATIONSNAME $DEFENESTRATIONSCITY site:metafilter.com I do not get a result. This indicates that the content of the page is not what is causing the search result, which aligns with not having a cached result. There is some other pointer ...I have seen this before when searching for someone using a
site:facebook.com and getting them, yet with a different last name because they got married ... with absolutely no hint of the old last name in the HTML of the Facebook page. Yeah. There's some kind of pointer out there.
posted by adipocere at 1:12 PM on February 22, 2011
I'm really not thrilled about this.
adipocere: "There is some other pointer ..."
Inbound links? I shouldn't have any to my profile.
posted by zarq at 1:15 PM on February 22, 2011
adipocere: "There is some other pointer ..."
Inbound links? I shouldn't have any to my profile.
posted by zarq at 1:15 PM on February 22, 2011
I just remembered that I link my MeFI profile on my Facebook page. My Facebook page is googleable in that you can find me if you google my name, but private in that you can't access much information, such as the sites I link, so it's still somewhat confusing.
posted by defenestration at 1:16 PM on February 22, 2011
posted by defenestration at 1:16 PM on February 22, 2011
It's pretty normal and unsurprising that Googling your username would bring up your MeFi profile, right? Because your username links directly to your profile from tons of indexed pages. It's the Googling of real names turning up MeFi profiles that should be of more concern, although that could be the result of the same thing, somebody linking to your MeFi profile using linktext of your real name, right?
posted by Gator at 1:22 PM on February 22, 2011
posted by Gator at 1:22 PM on February 22, 2011
Luckily for me, my username has a couple (thousand) links to it, and for my actual name, it would seem there's a rather popular dentist (or one who's really mastered SEO). It would also seem that someone of the same name got busted recently, and I can see his mugshot.
But yeah, it would be nice if here wasn't all over there.
posted by Ghidorah at 1:24 PM on February 22, 2011
But yeah, it would be nice if here wasn't all over there.
posted by Ghidorah at 1:24 PM on February 22, 2011
if it helps.. googling:
royalsong site:metafilter.com
royal song site:metafilter.com
does not bring up my profile, only posts I've posted in.
posted by royalsong at 1:24 PM on February 22, 2011
royalsong site:metafilter.com
royal song site:metafilter.com
does not bring up my profile, only posts I've posted in.
posted by royalsong at 1:24 PM on February 22, 2011
Well, there is that explicit NO FOLLOW thing, though.
That's only once you get to the actual profile page. Your username links directly to your profile on every single post and comment you make, with no NOFOLLOW attached to those links.
posted by Gator at 1:26 PM on February 22, 2011
That's only once you get to the actual profile page. Your username links directly to your profile on every single post and comment you make, with no NOFOLLOW attached to those links.
posted by Gator at 1:26 PM on February 22, 2011
What's interesting about that search result is that there is no cached version of it listed, yet there are no cache directives on the HTML source.
Yeah, that's the thing. The way the "indexed" profiles show up is so out of wack with how a normally indexed page would show up (no title listed, no summary, no cache, no ability to search on much of the profile page content) that it really feels like this is an artifact of some other thing Google is doing.
It's possible this is some sort of side-effect of all the crazy blog-oriented optimization they've been doing over the years; maybe some sort of fucked-up downside to the fact that Metafilter gets indexed super promptly in terms of new posts and RSS stuff, say. But I don't even know where to start as far as getting answers to any of this.
It's pretty normal and unsurprising that Googling your username would bring up your MeFi profile, right?
We explicitly tell Google not to index those pages, according to the standard directives used to tell search engines such things. Google is generally very good about such stuff; it should basically refuse to acknowledge that the pages even exist by normal expectations.
posted by cortex (staff) at 1:29 PM on February 22, 2011
Yeah, that's the thing. The way the "indexed" profiles show up is so out of wack with how a normally indexed page would show up (no title listed, no summary, no cache, no ability to search on much of the profile page content) that it really feels like this is an artifact of some other thing Google is doing.
It's possible this is some sort of side-effect of all the crazy blog-oriented optimization they've been doing over the years; maybe some sort of fucked-up downside to the fact that Metafilter gets indexed super promptly in terms of new posts and RSS stuff, say. But I don't even know where to start as far as getting answers to any of this.
It's pretty normal and unsurprising that Googling your username would bring up your MeFi profile, right?
We explicitly tell Google not to index those pages, according to the standard directives used to tell search engines such things. Google is generally very good about such stuff; it should basically refuse to acknowledge that the pages even exist by normal expectations.
posted by cortex (staff) at 1:29 PM on February 22, 2011
Ah okay, back up. If I ask google:
royalsong site:metafilter.com -ask -metatalk
then my profile comes up.. and only when I type that. If i don't tell it to exclude -metatalk.. only two pages pop up all on metatalk and my one lonely comment on the blue. My profile page isn't in that listing.
posted by royalsong at 1:30 PM on February 22, 2011
royalsong site:metafilter.com -ask -metatalk
then my profile comes up.. and only when I type that. If i don't tell it to exclude -metatalk.. only two pages pop up all on metatalk and my one lonely comment on the blue. My profile page isn't in that listing.
posted by royalsong at 1:30 PM on February 22, 2011
I just tried googling my real name and the metafilter site but didn't get anything.
posted by hydrobatidae at 1:30 PM on February 22, 2011
posted by hydrobatidae at 1:30 PM on February 22, 2011
When I google my metafilter username there is nothing about me on the first four pages search results. No posts, no profile, nothing.
I do not link to metafilter from elsewhere, although I do have a couple of links to elsewhere that I have posted on metafilter.
posted by bukvich at 1:38 PM on February 22, 2011
I do not link to metafilter from elsewhere, although I do have a couple of links to elsewhere that I have posted on metafilter.
posted by bukvich at 1:38 PM on February 22, 2011
Let's see, so I can find some six-digit userids in Google's search results with a
But it isn't caching. Why?
The
Google might not necessarily need a hyperlink as a pointer, just some kind of internal storage knowing that "Joe Smith" is associated with these two different pages, like knowing that this "firstname1 lastname1" is really "firstname1 lastname2" because she needed a name for her exotic dancing (yeah, I've seen this).
What I wouldn't give to trawl through some BigTable to see why.
posted by adipocere at 1:39 PM on February 22, 2011
site:www.metafilter.com/user. So this isn't a one-time whoopsie where Google accidentally crawled a few years back and just never scrubbed the old data. This is ongoing.But it isn't caching. Why?
The
title of the page is kept, and yet searching by intitle isn't pulling stuff up. Google might not necessarily need a hyperlink as a pointer, just some kind of internal storage knowing that "Joe Smith" is associated with these two different pages, like knowing that this "firstname1 lastname1" is really "firstname1 lastname2" because she needed a name for her exotic dancing (yeah, I've seen this).
What I wouldn't give to trawl through some BigTable to see why.
posted by adipocere at 1:39 PM on February 22, 2011
Google is generally very good about such stuff; it should basically refuse to acknowledge that the pages even exist by normal expectations.
Maybe something to do with this? "Note that because we have to crawl your page in order to see the noindex meta tag, there's a small chance that Googlebot won't see and respect the noindex meta tag. If your page is still appearing in results, it's probably because we haven't crawled your site since you added the tag. (Also, if you've used your robots.txt file to block this page, we won't be able to see the tag either.)" (Emphasis mine.)
posted by Gator at 1:43 PM on February 22, 2011 [1 favorite]
Maybe something to do with this? "Note that because we have to crawl your page in order to see the noindex meta tag, there's a small chance that Googlebot won't see and respect the noindex meta tag. If your page is still appearing in results, it's probably because we haven't crawled your site since you added the tag. (Also, if you've used your robots.txt file to block this page, we won't be able to see the tag either.)" (Emphasis mine.)
posted by Gator at 1:43 PM on February 22, 2011 [1 favorite]
I'm really glad I share a name with a famous author. I'm buried waaaaay down the google results.
posted by brand-gnu at 1:59 PM on February 22, 2011
posted by brand-gnu at 1:59 PM on February 22, 2011
When I google my real name, no MeFi links come up at all.
When I google my username, my profile page is the very first result.
posted by marsha56 at 2:03 PM on February 22, 2011
When I google my username, my profile page is the very first result.
posted by marsha56 at 2:03 PM on February 22, 2011
note: Items with a highlighted background are only shown to other logged-in members, and won't be seen by the general public or search engines
But can I still get a hug?
posted by infini at 2:03 PM on February 22, 2011
But can I still get a hug?
posted by infini at 2:03 PM on February 22, 2011
Check it, so if I search for $DEFENESTRATIONSNAME $DEFENESTRATIONSCITY site:metafilter.com I do not get a result. This indicates that the content of the page is not what is causing the search result, which aligns with not having a cached result. There is some other pointer ...
Right, and searching for site:metafilter.com/user +"New York" brings up this profile, even though "New York" doesn't appear on the profile page at all.
posted by burnmp3s at 2:03 PM on February 22, 2011
Right, and searching for site:metafilter.com/user +"New York" brings up this profile, even though "New York" doesn't appear on the profile page at all.
posted by burnmp3s at 2:03 PM on February 22, 2011
I don't have anything linked to my profile but sure enough Teh Goog has seen it.
Every MetaFilter thread you've participated in links to your profile. So Google has seen it many, many times. It could be that those numerous mentions of your [your username] = [this page] overrides Google's [don't index this] rule but not its [don't cache this] rule. Pure speculation on my part, but it kind of makes sense.
Yeah, we have the robots.txt set for profile pages and the explicit
posted by pb (staff) at 2:04 PM on February 22, 2011
Every MetaFilter thread you've participated in links to your profile. So Google has seen it many, many times. It could be that those numerous mentions of your [your username] = [this page] overrides Google's [don't index this] rule but not its [don't cache this] rule. Pure speculation on my part, but it kind of makes sense.
Yeah, we have the robots.txt set for profile pages and the explicit
NOINDEX set for each page as per their instructions. Interesting Gator, maybe we should remove the robots.txt rule.posted by pb (staff) at 2:04 PM on February 22, 2011
In keeping the
So maybe there are additional levels besides just an index and a cache, like metadata. That would explain how the
At some of the insider conferences I've been to, which tend to track Google pretty closely, there's some pretty pointed talk about Google having a good idea who you are, as an entity, even though that is obscured through much abstraction. All it has to do is know that $DEFENESTRATIONENTITY has an association with a certain page about which it has metadata for the results to show.
Tonight I think I am going to try to tease out what metadata Google is keeping and test this on some profiles. Maybe compare with Bing, too.
posted by adipocere at 2:14 PM on February 22, 2011
title, you know it has at least seen the page and kept some metadata about it, but if it had truly kept the contents of the page indexed, adding a city search term the defenestration-search shouldn't make the results drop to zero. So maybe there are additional levels besides just an index and a cache, like metadata. That would explain how the
title is kept even if the page isn't properly indexed.At some of the insider conferences I've been to, which tend to track Google pretty closely, there's some pretty pointed talk about Google having a good idea who you are, as an entity, even though that is obscured through much abstraction. All it has to do is know that $DEFENESTRATIONENTITY has an association with a certain page about which it has metadata for the results to show.
Tonight I think I am going to try to tease out what metadata Google is keeping and test this on some profiles. Maybe compare with Bing, too.
posted by adipocere at 2:14 PM on February 22, 2011
Searching 'site:metafilter.com "my real name"' brings up a single result - this page from 2007 which I commented in, but the page doesn't contain either my first or last name. The result shows a cached copy, but clicking on what should be a link to that takes me to the Google home page and breaks the back button.
Curious.
posted by dg at 2:16 PM on February 22, 2011
Curious.
posted by dg at 2:16 PM on February 22, 2011
Maybe compare with Bing, too.
Bing would probably be right behind you comparing itself to your work
as an aside, the beanplater within is musing upon beauty of teh OP's username and the topic at hand, is eponysterical the right word or should there be another one for nuance?
posted by infini at 2:19 PM on February 22, 2011 [1 favorite]
Bing would probably be right behind you comparing itself to your work
as an aside, the beanplater within is musing upon beauty of teh OP's username and the topic at hand, is eponysterical the right word or should there be another one for nuance?
posted by infini at 2:19 PM on February 22, 2011 [1 favorite]
And finally, this comment looks funny.
outta here now
posted by infini at 2:23 PM on February 22, 2011
outta here now
posted by infini at 2:23 PM on February 22, 2011
I searched for my username, limited to metafilter.com, and my profile page didn't come up in the first twelve pages. Nor was is in the two results that came back from searching for my real name.
Just another datapoint...
posted by nickmark at 2:27 PM on February 22, 2011
Just another datapoint...
posted by nickmark at 2:27 PM on February 22, 2011
As Gator mentioned, robots.txt prevents Google from spidering a page. The noindex meta tag prevents the page from appearing in search results. The problem here is that Google knows about the profile pages even though it has never crawled them. The solution is to modify your robots.txt to allow googlebot to read the profile pages. Then it will see the noindex meta tag and drop the page from the search results.
Another way to do this is to keep your robots.txt the way it is and use Google's URL removal tool. An explicit removal request along with a robots.txt block will completely prevent a URL from appearing in search results.
posted by ryanrs at 2:31 PM on February 22, 2011
Another way to do this is to keep your robots.txt the way it is and use Google's URL removal tool. An explicit removal request along with a robots.txt block will completely prevent a URL from appearing in search results.
posted by ryanrs at 2:31 PM on February 22, 2011
I searched for my username, limited to metafilter.com, and my profile page didn't come up in the first twelve pages.
A direct search finds it for your user name (twice for some reason). Not for your real name though.
posted by burnmp3s at 2:32 PM on February 22, 2011
A direct search finds it for your user name (twice for some reason). Not for your real name though.
posted by burnmp3s at 2:32 PM on February 22, 2011
A direct search finds it for your user name (twice for some reason).
Huh. Sure enough.
Not for your real name though.
Well... kinda. ;)
posted by nickmark at 2:35 PM on February 22, 2011 [1 favorite]
Huh. Sure enough.
Not for your real name though.
Well... kinda. ;)
posted by nickmark at 2:35 PM on February 22, 2011 [1 favorite]
Hmm. Looks like cortex outed me to the Google!
posted by misterbrandt at 2:39 PM on February 22, 2011
posted by misterbrandt at 2:39 PM on February 22, 2011
Their documentation is a bit contradictory. In one place they say robots.txt is the preferred method of keeping pages out of their index, with the
I just updated our robots.txt file to allow google to crawl user pages which—according to their documentation—will let them see the
posted by pb (staff) at 2:42 PM on February 22, 2011
NOCACHE directive an alternate. In another place they say the robots.txt directive can interfere with keeping a page out of their index.I just updated our robots.txt file to allow google to crawl user pages which—according to their documentation—will let them see the
NOCACHE directive we have set on the page. Let's give that time to work its way through Google's updates and see if that helps.posted by pb (staff) at 2:42 PM on February 22, 2011
Uh, don't you mean noindex ?
Robots.txt removes the page. Nocache removes the cached copy. Noindex removes the URL (and therefore the other two as well).
posted by ryanrs at 2:46 PM on February 22, 2011
Robots.txt removes the page. Nocache removes the cached copy. Noindex removes the URL (and therefore the other two as well).
posted by ryanrs at 2:46 PM on February 22, 2011
Sorry, yeah, I misspoke. We have
posted by pb (staff) at 2:48 PM on February 22, 2011
noindex on profile pages.posted by pb (staff) at 2:48 PM on February 22, 2011
I think pb has it. From that first "one place":
If the page still exists but you don't want it to appear in search results, use robots.txt to prevent Google from crawling it. Note that in general, even if a URL is disallowed by robots.txt we may still index the page if we find its URL on another site. However, Google won't index the page if it's blocked in robots.txt and there's an active removal request for the page.
Note the "AND there's an active removal request" (detailed below that). Since you guys don't want to have to request every profile be removed manually, you want the second option:
Alternatively, you can use a noindex meta tag. When we see this tag on a page, Google will completely drop the page from our search results, even if other pages link to it. This is a good solution if you don't have direct access to the site server. (You will need to be able to edit the HTML source of the page).
posted by wildcrdj at 2:49 PM on February 22, 2011
If the page still exists but you don't want it to appear in search results, use robots.txt to prevent Google from crawling it. Note that in general, even if a URL is disallowed by robots.txt we may still index the page if we find its URL on another site. However, Google won't index the page if it's blocked in robots.txt and there's an active removal request for the page.
Note the "AND there's an active removal request" (detailed below that). Since you guys don't want to have to request every profile be removed manually, you want the second option:
Alternatively, you can use a noindex meta tag. When we see this tag on a page, Google will completely drop the page from our search results, even if other pages link to it. This is a good solution if you don't have direct access to the site server. (You will need to be able to edit the HTML source of the page).
posted by wildcrdj at 2:49 PM on February 22, 2011
As I recall, googlebot doesn't check robots.txt very often. So it may be a while before it sees the changes. (Your webserver logs should tell you how often it polls robots.txt for changes.)
posted by ryanrs at 2:50 PM on February 22, 2011
posted by ryanrs at 2:50 PM on February 22, 2011
A direct search finds it for your user name (twice for some reason). Not for your real name though.
The reason it comes up twice is because the links are /user.mefi/ and /user/.
posted by knapah at 2:51 PM on February 22, 2011
The reason it comes up twice is because the links are /user.mefi/ and /user/.
posted by knapah at 2:51 PM on February 22, 2011
The processing pipeline looks like this:
[robots.txt allows?] -> "the crawl" -> [no noindex?] -> "the index"
Now that the robots.txt block is removed, our profile pages will be stored in "the crawl" (but not in copied into the index). I, for one, am pleased that my favorite count will be preserved for all eternity. (Google has never deleted a crawl.)
posted by ryanrs at 3:02 PM on February 22, 2011
[robots.txt allows?] -> "the crawl" -> [no noindex?] -> "the index"
Now that the robots.txt block is removed, our profile pages will be stored in "the crawl" (but not in copied into the index). I, for one, am pleased that my favorite count will be preserved for all eternity. (Google has never deleted a crawl.)
posted by ryanrs at 3:02 PM on February 22, 2011
I did a search for my username; there's a picture of little e and me at the 10th Meetup, something about how to tell a pterodactyl's sex (mine is in my profile. Shortcut!) and something with the description "Nitrous-Injected Merkin Bandit of Yesteryear" which doesn't sound like me but I guess you never know?
posted by Mrs. Pterodactyl at 3:24 PM on February 22, 2011
posted by Mrs. Pterodactyl at 3:24 PM on February 22, 2011
Back before profiles were excluded I ranked higher than Jagger for Mick.
posted by Mick at 3:45 PM on February 22, 2011 [1 favorite]
posted by Mick at 3:45 PM on February 22, 2011 [1 favorite]
Searching for my user name turns up 1) my MeFi profile; 2) a post by flapjax where I'm the first commenter, 3) a post by alopez where I'm one of the later commenters; 4) my post on DnB, and then two links to my user name on the unofficial wiki, then it gets more random. I wonder if those other threads were broadly popular posts in terms of sites linking to them from elsewhere.
I don't include my personal information on my MeTa profile (the location is a pleasant hillside near where I used to live), though there's enough personal stuff accessable by digging a few layers down that I've been emailed about a post I made by a non-MeFite. It was weird, until I found where I had put my email address.
It's links, man. Links all the way down.
posted by filthy light thief at 3:50 PM on February 22, 2011
I don't include my personal information on my MeTa profile (the location is a pleasant hillside near where I used to live), though there's enough personal stuff accessable by digging a few layers down that I've been emailed about a post I made by a non-MeFite. It was weird, until I found where I had put my email address.
It's links, man. Links all the way down.
posted by filthy light thief at 3:50 PM on February 22, 2011
I'm guessing that google has changed something recently. Better cross-indexing? I have no idea really, but the search results for my username are much different than usual and much better actually. For example in the first couple of pages there are links to old forgotten projects that I haven't seen on google for quite a while.
posted by snsranch at 3:58 PM on February 22, 2011
posted by snsranch at 3:58 PM on February 22, 2011
Yeah, if you Google my real name, my Metafilter profile shows up.
posted by Brandon Blatcher at 4:08 PM on February 22, 2011 [1 favorite]
posted by Brandon Blatcher at 4:08 PM on February 22, 2011 [1 favorite]
Googling flabdablet site:metafilter.com gets me my profile page, but it only shows my personal details if I'm logged in to mefi on the browser involved, and there's no cached link available on the search page. I'm happy with that.
posted by flabdablet at 4:25 PM on February 22, 2011
posted by flabdablet at 4:25 PM on February 22, 2011
Defenestration: I just remembered that I link my MeFI profile on my Facebook page
Why keep blaming teh googles, people? Facebook, known blabber of sensitive information, seems like a more likely leak-point to me. I'm really creeped out at this point by the random sites that will grab my fb id and 'log me in' when i have the misfortune to surf them when my fb page is up.
posted by Rube R. Nekker at 4:35 PM on February 22, 2011
Why keep blaming teh googles, people? Facebook, known blabber of sensitive information, seems like a more likely leak-point to me. I'm really creeped out at this point by the random sites that will grab my fb id and 'log me in' when i have the misfortune to surf them when my fb page is up.
posted by Rube R. Nekker at 4:35 PM on February 22, 2011
Alternatively, you can use a noindex meta tag. When we see this tag on a page, Google will completely drop the page from our search results, even if other pages link to it.
Interesting, and I guess this makes sense. Just find it unfortunate that now google is pulling in all that user-specific data; they may not be using it in search results, but I imagine it sure as heck is being used for advertising and other purposes.
posted by inigo2 at 4:37 PM on February 22, 2011
Interesting, and I guess this makes sense. Just find it unfortunate that now google is pulling in all that user-specific data; they may not be using it in search results, but I imagine it sure as heck is being used for advertising and other purposes.
posted by inigo2 at 4:37 PM on February 22, 2011
I just want to state for the record that I do not own a nitrous-burning trans-am.
posted by Devils Rancher at 4:40 PM on February 22, 2011
posted by Devils Rancher at 4:40 PM on February 22, 2011
Google is generally very good about such stuff; it should basically refuse to acknowledge that the pages even exist by normal expectations.
But it hasn't really been like that for years. I've told them again and again to remove my domain names from their index, using every developer tool and combination of robots.txt and metatags I can find. I even blocked googlebot via .htaccess, successfully for a while until they changed their IP (again!) and I gave up. Sometimes I made it go away for a short time but never for very long. If you look up the stripped down domain url it brings up a listing for the domain home page, listed without cache or any other details (except page title) just like my profile page is now coming up when you google my username. I don't know if each domain appears when you look up specific search terms rather than the url because it's likely many pages deep and I don't care enough to wade through looking for it, but I expect it does. It changed sometime many years back, I don't remember when but probably six to eight years ago, when google got more aggressive about indexing in general.
So the refusing to acknowledge it exists thing isn't the case, and hasn't been for a long time. If they know about the site they will list it in their search results, just without the extra details you normally see. Maybe the more recent dev tools let you drop listings more permanently and easily, I haven't looked in a while. For my domain names I don't care, because if someone is google-ing the url they clearly know it exists and can just go straight there plus there is no content anyway. Having my username bringing up my profile page isn't really a big deal either because searching on that would otherwise bring up pages in other parts of metafilter I've posted in anyway. But I can see why people are concerned that other details on their profile pages are being indexed and I hope you're more successful than I was in getting google to stop indexing things you clearly don't want indexed.
posted by shelleycat at 4:52 PM on February 22, 2011
But it hasn't really been like that for years. I've told them again and again to remove my domain names from their index, using every developer tool and combination of robots.txt and metatags I can find. I even blocked googlebot via .htaccess, successfully for a while until they changed their IP (again!) and I gave up. Sometimes I made it go away for a short time but never for very long. If you look up the stripped down domain url it brings up a listing for the domain home page, listed without cache or any other details (except page title) just like my profile page is now coming up when you google my username. I don't know if each domain appears when you look up specific search terms rather than the url because it's likely many pages deep and I don't care enough to wade through looking for it, but I expect it does. It changed sometime many years back, I don't remember when but probably six to eight years ago, when google got more aggressive about indexing in general.
So the refusing to acknowledge it exists thing isn't the case, and hasn't been for a long time. If they know about the site they will list it in their search results, just without the extra details you normally see. Maybe the more recent dev tools let you drop listings more permanently and easily, I haven't looked in a while. For my domain names I don't care, because if someone is google-ing the url they clearly know it exists and can just go straight there plus there is no content anyway. Having my username bringing up my profile page isn't really a big deal either because searching on that would otherwise bring up pages in other parts of metafilter I've posted in anyway. But I can see why people are concerned that other details on their profile pages are being indexed and I hope you're more successful than I was in getting google to stop indexing things you clearly don't want indexed.
posted by shelleycat at 4:52 PM on February 22, 2011
> Yeah, if you Google my real name, my Metafilter profile shows up.
Wow, that is cool. When I google Al Ansari Faiz Muhammad I do get your metafilter account.
What? You people thought his real name is Brandon Blatcher? Yeah right. Why do you think his friends call him Fez?
posted by cjorgensen at 5:07 PM on February 22, 2011 [2 favorites]
Wow, that is cool. When I google Al Ansari Faiz Muhammad I do get your metafilter account.
What? You people thought his real name is Brandon Blatcher? Yeah right. Why do you think his friends call him Fez?
posted by cjorgensen at 5:07 PM on February 22, 2011 [2 favorites]
Wow. My username comes up fifth in the Google results. I was all excited about being the top individual among the hits, but #4 is a redditor, apparently.
posted by Navelgazer at 5:50 PM on February 22, 2011
posted by Navelgazer at 5:50 PM on February 22, 2011
Yeah, the first hit for my username is my profile. The next hits are posts of mine to various MeFi sites.
Weird.
posted by valkyryn at 6:17 PM on February 22, 2011
Weird.
posted by valkyryn at 6:17 PM on February 22, 2011
cortex, if you google his real name (as listed in his profile) thusly: "real name" site:metafilter.com, it comes up with a single result: his profile.
dersins mcpoopypants site:metafilter.com
teh google, it does nothing.
posted by special-k at 7:23 PM on February 22, 2011
dersins mcpoopypants site:metafilter.com
teh google, it does nothing.
posted by special-k at 7:23 PM on February 22, 2011
"cjorgensen" not in quotes is my metafilter profile. In quotes it's one of my twitter accounts.
This is mostly true story, but the names have been changed to protect the innocent: I'm the majority of Christopher Jorgensens on the first several pages of google. It goes up if you add in my middle initial. I am also the majority of hits for cjorgensen. These go up if you put them in quotes so you eliminate the results that come up with "c" and "jorgensen" (as opposed to a singular search term like my username).
Anyway, I get this email from this guy named Christopher Jorgensen offering me $3,000 for one of my websites. At the time it was my primary blog and it would have been an effort to port the site to a new URL, but that seemed like real money to me. But I used google to google myself (more of less) and had to look until about page 20 on google for this particular Christopher Jorgensen (remember names have been changed!). And he's an architect or lawyer or some other highly paid professional. In fact, he's one of them partners in a firm. He's name is on their website.
I do some other searches and this guy is nearly invisible. I can't find a linkd in page, a youtube page, a facebook page, etc. So it dawns on me, he doesn't want my site for himself. He's just tired of people googling him and finding me. So I figure $30,000 can't be much to this guy, so I counter offer.
I still have the email somewhere where he calls me an asshole and tells me my name isn't worth that much.
p.s. I am not the cjorgensen that paints.
posted by cjorgensen at 7:51 PM on February 22, 2011
This is mostly true story, but the names have been changed to protect the innocent: I'm the majority of Christopher Jorgensens on the first several pages of google. It goes up if you add in my middle initial. I am also the majority of hits for cjorgensen. These go up if you put them in quotes so you eliminate the results that come up with "c" and "jorgensen" (as opposed to a singular search term like my username).
Anyway, I get this email from this guy named Christopher Jorgensen offering me $3,000 for one of my websites. At the time it was my primary blog and it would have been an effort to port the site to a new URL, but that seemed like real money to me. But I used google to google myself (more of less) and had to look until about page 20 on google for this particular Christopher Jorgensen (remember names have been changed!). And he's an architect or lawyer or some other highly paid professional. In fact, he's one of them partners in a firm. He's name is on their website.
I do some other searches and this guy is nearly invisible. I can't find a linkd in page, a youtube page, a facebook page, etc. So it dawns on me, he doesn't want my site for himself. He's just tired of people googling him and finding me. So I figure $30,000 can't be much to this guy, so I counter offer.
I still have the email somewhere where he calls me an asshole and tells me my name isn't worth that much.
p.s. I am not the cjorgensen that paints.
posted by cjorgensen at 7:51 PM on February 22, 2011
oh, and doing an image search on my username brings up... interesting things. A little ways down you get some meetup photos of me. Interestingly you do not get my user photo, but you do get Miss Lynnster's.
posted by Navelgazer at 8:05 PM on February 22, 2011
posted by Navelgazer at 8:05 PM on February 22, 2011
I just googled my name on the site, and happily my profile doesn't come up, but I did find some old threads discussing me in 2002, long before I became a member, which is a bit of a strange feeling.
posted by immlass at 8:14 PM on February 22, 2011
posted by immlass at 8:14 PM on February 22, 2011
As a data point (and by now maybe a useless one), searching for "myusername site:metafilter.com" doesn't return my profile, and searching "myrealname site:metafilter.com" returns no hits, despite my real name being on my profile. "myemail site:metafilter.com" gets nothing either. For me, at least, the process seems to be working.
Is it because I'm low-profile compared to a lot of the people posting in this thread? Because I have no outbound links on my profile page? Because I don't link my Metafilter account to my Facebook or Twitter accounts (though they do use the same email address)? Because Google hiccups at searching for a misspelling of an actual English word? Unknown.
For the curious, I'm not any of the myrealnames that show up in the first several pages of Google results. This is about the only place on the web that I use my real name.
posted by penduluum at 8:24 PM on February 22, 2011
Is it because I'm low-profile compared to a lot of the people posting in this thread? Because I have no outbound links on my profile page? Because I don't link my Metafilter account to my Facebook or Twitter accounts (though they do use the same email address)? Because Google hiccups at searching for a misspelling of an actual English word? Unknown.
For the curious, I'm not any of the myrealnames that show up in the first several pages of Google results. This is about the only place on the web that I use my real name.
posted by penduluum at 8:24 PM on February 22, 2011
Yay for having a boring name IRL and a boring username! Google autocorrects to martyr.
Meanwhile, if I ever publish, no one will ever find my papers. Sigh.
posted by maryr at 8:59 PM on February 22, 2011 [1 favorite]
Meanwhile, if I ever publish, no one will ever find my papers. Sigh.
posted by maryr at 8:59 PM on February 22, 2011 [1 favorite]
I did the "google your real name specific to site:metafilter.com" thing and was surprised to find that I am a famous jazz musician.
Yes, my real name is Benny Goodman.
posted by Curious Artificer at 5:16 AM on February 23, 2011
Yes, my real name is Benny Goodman.
posted by Curious Artificer at 5:16 AM on February 23, 2011
Oh, false alarm; I'm just an author.
posted by Curious Artificer at 5:17 AM on February 23, 2011
posted by Curious Artificer at 5:17 AM on February 23, 2011
It's not just links, I don't think - I'm pretty sure I have links to my MetaFilter profile out there, but my profile isn't in the first four pages of Google results when searching for my username (meanwhile, my Twitter page is third, after GWU's School of Media and Public Affairs and the San Miguel Power Association.)
When confining the search to MetaFilter, I get hits in threads where people quote me, then the stuff I posted myself.
posted by SMPA at 6:32 AM on February 23, 2011
When confining the search to MetaFilter, I get hits in threads where people quote me, then the stuff I posted myself.
posted by SMPA at 6:32 AM on February 23, 2011
The answer to this questino in the FAQ may need updating:
What is the etiquette concerning member's or non-member's personal details and profile information?
User profile pages on MetaFilter are not indexed by Google...
posted by mediareport at 6:35 AM on February 23, 2011
What is the etiquette concerning member's or non-member's personal details and profile information?
User profile pages on MetaFilter are not indexed by Google...
posted by mediareport at 6:35 AM on February 23, 2011
this questino
The LHC has detected the elementary interrogative particle at last!
posted by aught at 6:45 AM on February 23, 2011 [4 favorites]
The LHC has detected the elementary interrogative particle at last!
posted by aught at 6:45 AM on February 23, 2011 [4 favorites]
Which advances the ongoing theory in particle physiosyntactics of a corresponding "exclamino". Proof of such would be exciting in its own right, but would also make more plausible the notion of an interaction between the two that would be so violent that everybody at CERN would have to run from the bang in terror.
posted by cortex (staff) at 6:59 AM on February 23, 2011 [7 favorites]
posted by cortex (staff) at 6:59 AM on February 23, 2011 [7 favorites]
I tried the site specific search on my last name (which is very unusual so it would not be odd at all if I was the only one on the site) and all I got back was this short AskMe on low salt ham, which is pretty funny.
posted by nanojath at 7:39 AM on February 23, 2011
posted by nanojath at 7:39 AM on February 23, 2011
I have a very plain first, middle, and last name (at least in the US) and though I'm very proud of each of my names since they each have a nice personal meaning, I am glad to have such ordinary names that a Google search finds nothing to do with me in the first several pages. If there was a result linked to me based on my MeFi profile, I would also not be too worried either since I have made no attempt to anonymize myself here. I am what I am, for better or worse.
I would be worried if people who were attempting to anonymize their MeFi profiles by changing a name, location, or personal details were outed by Google. MeFi is great because of the combined anonyminity and openness, not because of one or the other.
posted by Clinging to the Wreckage at 7:57 AM on February 23, 2011
I would be worried if people who were attempting to anonymize their MeFi profiles by changing a name, location, or personal details were outed by Google. MeFi is great because of the combined anonyminity and openness, not because of one or the other.
posted by Clinging to the Wreckage at 7:57 AM on February 23, 2011
> Oh, false alarm; I'm just an author.
I ran your metafilter writings through some complex Al Gore Rhythms and it says you write like Dan Brown.
posted by cjorgensen at 8:14 AM on February 23, 2011
I ran your metafilter writings through some complex Al Gore Rhythms and it says you write like Dan Brown.
posted by cjorgensen at 8:14 AM on February 23, 2011
If you Google "Mayor Curley" (without the quotes), my profile is the third result. Which seems like a shortcoming on Google's part seeing as how my nick is simply the popular form of address for a very well-documented historical figure. And I'm just some dude making comments on the Internet.
posted by Mayor Curley at 8:25 AM on February 23, 2011
posted by Mayor Curley at 8:25 AM on February 23, 2011
Huh, I can't find my MeFi profile on Google, using my nick, my old nick or my real name or any combination thereof. Am I doing something wrong or am I doing something right?
posted by oneswellfoop at 10:03 AM on February 23, 2011
posted by oneswellfoop at 10:03 AM on February 23, 2011
My username is the third Google hit (the first two are my Flickr account). My real name does not link to Metafilter in the first six pages of results.
posted by anastasiav at 11:22 AM on February 23, 2011
posted by anastasiav at 11:22 AM on February 23, 2011
I looked back five pages and my profile does not come up with or without quotes. Lots of threads that I posted in, but no profile. Perhaps it's because I use only my first name and/or it's a "boring name" like maryr mentioned.
It's kind of sad. I used to be deborah #2 or #3 when I was still a prolific blogger.
posted by deborah at 12:38 PM on February 23, 2011
It's kind of sad. I used to be deborah #2 or #3 when I was still a prolific blogger.
posted by deborah at 12:38 PM on February 23, 2011
When I google my user name my profile comes up, but holy christ on a cracker, this comes up too!
posted by MexicanYenta at 8:48 PM on February 23, 2011
posted by MexicanYenta at 8:48 PM on February 23, 2011
i just googled my username - my metafilter profile comes up 2nd - not that there's any real info there - my adequacy.org profile comes in 1st - 3rd is my music on music.metafilter
and then there's a bunch of pest control companies, some of which claim to be "pyramid termite and pest control", although they all seem to be located in apartment or duplex communities and don't really exist
I.WILL.NOT.BE.EXTERMINATED!!!
posted by pyramid termite at 9:47 PM on February 23, 2011
and then there's a bunch of pest control companies, some of which claim to be "pyramid termite and pest control", although they all seem to be located in apartment or duplex communities and don't really exist
I.WILL.NOT.BE.EXTERMINATED!!!
posted by pyramid termite at 9:47 PM on February 23, 2011
I am not a robots.txt expert, but I think that Metafilter's is not correctly designed. What I gather to be the standard says:
posted by Flunkie at 5:19 AM on February 24, 2011
If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records.So the following part of Metafilter's robots.txt:
User-agent: Googlebot
Disallow: /*rss$
Disallow: /trash/
Disallow: /friendsfavorites/
Disallow: /favorited/User-agent: *
Disallow: /user.mefi/
Disallow: /username.mefi/
Disallow: /user/
Disallow: /username/
Disallow: /trash/
Disallow: /friendsfavorites/
Disallow: /favorited/
Crawl-delay: 5posted by Flunkie at 5:19 AM on February 24, 2011
Also, I can't help but notice the following comment at the top of the file:
posted by Flunkie at 5:30 AM on February 24, 2011
# robots txt v. 1.08 Feb 22 2011 (googlebot changes)posted by Flunkie at 5:30 AM on February 24, 2011
You're right, Flunkie. Based on the conversation in this thread, we updated our robots.txt file. Here's where I mentioned it: "I just updated our robots.txt file to allow google to crawl user pages which—according to their documentation—will let them see the NOINDEX directive we have set on the page. Let's give that time to work its way through Google's updates and see if that helps."
posted by pb (staff) at 7:37 AM on February 24, 2011
posted by pb (staff) at 7:37 AM on February 24, 2011
This was my favorite hit for searching site:metafilter.com for galadriel. (My profile didn't come up.)
As far as I know my dad has never shouted "Frodo Lives!" or attended a sci-fi con.
posted by galadriel at 10:40 AM on February 24, 2011
As far as I know my dad has never shouted "Frodo Lives!" or attended a sci-fi con.
posted by galadriel at 10:40 AM on February 24, 2011
Profile pages are still coming up for certain users if you search with
If anyone is bothered by their profile link showing up on Google, send me a MeFi Mail and I'll ask Google to remove the page explicitly. That should speed up the process.
posted by pb (staff) at 10:13 AM on March 1, 2011
site:metafilter.com [username]. Google takes at least few weeks to update their index after changes like this so I think it's too early to assess whether or not the robots.txt changes made a difference.If anyone is bothered by their profile link showing up on Google, send me a MeFi Mail and I'll ask Google to remove the page explicitly. That should speed up the process.
posted by pb (staff) at 10:13 AM on March 1, 2011
It's been a while now and I can't find profile pages with searches that were working before. It looks like our robots.txt change did the trick. Let me know if not and I'll ask for those individual pages to be removed.
posted by pb (staff) at 3:26 PM on March 15, 2011
posted by pb (staff) at 3:26 PM on March 15, 2011
You are not logged in, either login or create an account to post comments
posted by blue_beetle at 12:34 PM on February 22, 2011 [6 favorites]