Is Google indexing broken?

November 23, 2002 12:20 AM

When I wanted to do a search to find out if someone had linked to a given site in a FPP yet, I decided to try something different. I went to old google and put in this URL, which was the link in the most popular FPPof the day exactly two weeks ago, as a "who link's to you?" search.

Shouldn't google have indexed the thread, then, since it links back to the original article? Strange, that it doesn't do this for a link and lenghty thread on a topic that will have serious implications for everyone on the internet, specifically Americans? Has google dropped the ball? Is this all just a bunch of Tom-Foolery?
posted by Hammerikaner to MetaFilter-related at 12:20 AM (8 comments total)

Google hasn't visited MeFi recently, as is visible from their cache: http://216.239.39.100/search?q=cache:www.metafilter.com

In the past they did visit very dynamic sites a bit too often, looks like they've changed the algorithm and this time erred to updating too rarely.
posted by fvw at 12:26 AM on November 23, 2002

I am surprised Google still caches sites like MeFi at all, but once a quarter seems adequate, all things considered.
posted by mischief at 12:35 AM on November 23, 2002

MeFi was getting indexed too often by the googlebot so I think Matt spoke with Google and had them decrease the indexing frequency.
It's 4:00 AM right now, I tried looking for the thread but I couldn't find it.
posted by riffola at 1:03 AM on November 23, 2002

robots.txt isn't excluding any threads. curiously, it seems that it's ignored by google (since the first hit searching for my name is my mefi user page, which should be blocked; on the other hand it's not cached).
posted by andrew cooke at 2:50 AM on November 23, 2002

If your user page was indexed prior to the creation of that robots.txt file, and Matt hasn't specifically requested them to purge their index, I'm pretty sure that they will stay. I don't think they completely scrap their database every time they send the googlebot out, do they?

I had a similar experience when I told Google not to include jpegs from one of my sites in their image index. I tried just changing the robots.txt file, but had to specifically tell them to purge the files from their database, suing a very finicky (as I recall) online form on their site.
posted by crunchland at 6:24 AM on November 23, 2002

We're dying to know: did the form countersue?
posted by dhartung at 7:50 AM on November 23, 2002

pttht.
posted by crunchland at 9:25 AM on November 23, 2002

I'll translate: "Yes, and they've just collected a hefty fine plus my tongue."

We'll be passing around the hat to see if we can raise enough to buy crunchland a prosthetic. Dig deep, folks.
posted by cortex at 5:01 PM on November 24, 2002