What cookies do I need to set in order to have a console app grab my /usersnear/ page? February 6, 2008 1:08 PM   Subscribe

What cookies do I need to set in order to have a DIY app grab my metafilter.com/usersnear/ page?

I am interested in writing an application (probably a Windows Form app in C#) that will 1) pull my /usersnear/ page, 2) pull the Lat/Lon info for the nearby users, and then 3) make one of those funky Google Map API thingies...just for me.

I can only see the /usersnear/ page when logged in. Obviously a stateless app is not logged in. But if I pass the right cookies - copied from my local browser cookie jar - is it possible to authenticate on the fly and grab that first page? I see some obvious cookie candidates like USER_ID, USER_NAME and USER_PASS, but I'm not sure about the rest of them, which are necessary or not.

Possible? Anyone ever tried it? Is this folly?
posted by sidereal to MetaFilter-Related at 1:08 PM (32 comments total)

I'd go with chocolate chip.
posted by blue_beetle at 1:27 PM on February 6, 2008


Not supported in IE6.
posted by sidereal at 1:33 PM on February 6, 2008


I don't think you're going to get much assistance here without more details on why anyone would want to be able to do such a thing.
posted by potch at 1:35 PM on February 6, 2008


Haven't the foggiest, but as a sideways jaunt on this, the Mefi Google Earth .kml file contains all of this info in one place.
posted by cortex (staff) at 1:41 PM on February 6, 2008 [3 favorites]


You could always just write your app to log in, and store the resulting cookies.
posted by Malor at 1:55 PM on February 6, 2008


I did this kind of thing once when I wrote a .NET app to scrape listings from a torrent tracker. The method I used was what Malor suggested, to write code to actually log into the site, and then (preserving the session state) browse the site.

Most of it is built into .NET so all you really need to do is create a HttpWebRequest object and set the CookieContainer property to a new CookieContainer. When you do that, you can send the login post and the page get with the same HttpWebRequest and the cookies will be preserved. Check out the documentation for CookieContainer.
posted by burnmp3s at 2:04 PM on February 6, 2008 [1 favorite]


Well, it would take you about 30 seconds to test this for yourself, for instance by refreshing the page and seeing if it works with various different combinations of cookies. I suspect USER_KEY and USER_ID will be enough, and possibly also USER_NAME. But really, you didn't need to spoil over 24 Metatalk-free hours for something you could have figured out on your own.
posted by Burger-Eating Invasion Monkey at 2:07 PM on February 6, 2008


Wow, I thought I was on AskMe and flagged blue_beetle's comment as noise. Sorry dude.
posted by ODiV at 2:13 PM on February 6, 2008


No problem. Here, have a cookie.
posted by blue_beetle at 2:29 PM on February 6, 2008


A couple of minutes' experimenting with curl suggests that you need USER_TOKEN, USER_ID and USER_NAME.
posted by chrismear at 2:43 PM on February 6, 2008


cortex - thanks, that's awesome. I didn't know about that.

Malor, burnmp3s - I'll do that, thank you. Seems like the smartest approach.

BEIM - I wrote a simple app before posting, and all the various combinations of cookies that I tried didn't earn me the logged-in page, alas. Also tried searching like a good MeFite. No dice on that, though it may be that my search-fu is not so powerful. {{{hug}}}

Was there a more appropriate place to post this MeFi technical question?
posted by sidereal at 2:45 PM on February 6, 2008


Naw, this is the place to post it. You're welcome to pre-empt posting altogether by sending us email, of course. But BEIM is just struggling with the crushing disappointment that is a Metatalk quiet streak spoiled. As, alas, they must always be.

*hugs BEIM so tight*
posted by cortex (staff) at 2:47 PM on February 6, 2008


But really, you didn't need to spoil over 24 Metatalk-free hours for something you could have figured out on your own.

On the other hand, it does you no injury for your neighbor to ask what cookies he needs to set in order to have a DIY app grab his metafilter.com/usersnear/page. It neither picks your pocket nor breaks your leg.
posted by Atom Eyes at 2:51 PM on February 6, 2008 [2 favorites]


I have powers beyond reason here!
posted by sidereal at 3:10 PM on February 6, 2008


But really, you didn't need to spoil over 24 Metatalk-free hours for something you could have figured out on your own.

Maybe my blog is more popular than I thought, with people getting increasingly excited about how long it has been since I last posted something.

"It's been four months since he posted!"

"Bet he can't not post for another two weeks! God, this is so great!"

I had no idea. Guess this might just propel me to the top of the "A" list.
posted by maxwelton at 5:42 PM on February 6, 2008 [1 favorite]


If you continue to refrain from playing your cards right, well, by god, they'll be silently goggling at the absence of your name in letters ten feet high.
posted by cortex (staff) at 5:45 PM on February 6, 2008 [2 favorites]


<span style="font-style: fire; font-size: thirty feet high; location: the far side of the Quentulus Quazgar Mountains in the land of Sevorbeupstry on the planet of Preliumtarn, which orbits the star Zarss, which is located in the Grey Binding Fiefdoms of Saxaquine">WE APOLOGISE FOR THE INCONVENIENCE</span>
posted by Plutor at 5:51 PM on February 6, 2008 [1 favorite]


Did I mention that I have NO IDEA WHAT THE ANSWER IS?
posted by jessamyn (staff) at 7:03 PM on February 6, 2008 [1 favorite]


.NET CookieContainer. Dude.
posted by sidereal at 7:41 PM on February 6, 2008


We really ought not to be aiming for a day without meta, and discouraging people from making posts... how else are we going to reach the next quasi-significant round number to salivate and celebrate over? I mean, make up your minds, people. Do we rejoice over the 200 godzillionth post, or do we celebrate over NO posts? It's all so very, very schizoid.
posted by Dave Faris at 7:41 PM on February 6, 2008


Obviously, we just love zeros. A single non-zero digit? That's bad. A single digit with a whole lot of nice zeros after it? Excellent. A nice big zero by itself? Also excellent.
posted by ssg at 8:39 PM on February 6, 2008


I hear a lot of talk about zeroes, ssg, but I don't hear a lot of you getting off his lawn.
posted by cortex (staff) at 10:24 PM on February 6, 2008


This talk of cookies is jarring.
posted by Cranberry at 12:05 AM on February 7, 2008 [1 favorite]


The easy way is just to embed axWebBrowser to fill out the login form fo you and do a bit of screen scraping afterwards through the DOM. I use this for my own little project. I did it in C++ but it's even easier to do in C# from what I've seen.
posted by IronLizard at 4:52 AM on February 7, 2008


I don't want to make a separate thread for this. Can we have the ability to flag from the recent activity page?
posted by ODiV at 12:06 PM on February 7, 2008


Can we have the ability to flag from the recent activity page?

The recent activity page is already a clusterfuck of cohabitating cracnky code and I think it's a teetering house of sticks and duct tape currently so I think the answer to that, were you to ask it in a separate thread, would be no.
posted by jessamyn (staff) at 12:51 PM on February 7, 2008


So commenting from recent activity would be right out then. :)
posted by ODiV at 1:31 PM on February 7, 2008


I made you a cookie but I eated it.
posted by deborah at 1:40 PM on February 7, 2008


I'm not asking for a pony or anything, but it might be worth checking out how Yahoo Pipes deals with private data sources and support that:

http://blog.pipes.yahoo.com/2007/11/19/new-private-field-support-for-passwords-application-ids-and-more/

The industry appears to be converging on the use of application IDs or keys for this sort of thing -- mostly to obviate the need for storing passwords in application config files, I think. See Flickr's authentication framework:

http://www.flickr.com/services/api/auth.howto.desktop.html

Eventually this is how MetaFilter should handle user authentication for programmatic APIs.
posted by sdodd at 5:21 PM on February 7, 2008


Cookies aren't ideal for programmatic access.
posted by sdodd at 5:23 PM on February 7, 2008


But really, you didn't need to spoil over 24 Metatalk-free hours for something you could have figured out on your own.

If only there were some other way to avoid reading any Metatalk threads for 24 hours besides preventing tens of thousands of people from posting any.

I almost think there should be a way for individual users to determine whether they see any Metatalk threads in a given day, but I can't quite figure out how that would work. Maybe pb can whip something up.
posted by Mr. President Dr. Steve Elvis America at 12:07 AM on February 10, 2008 [1 favorite]


I lol'd.
posted by Burger-Eating Invasion Monkey at 7:38 AM on February 10, 2008


« Older Drawn MetaFilter comments   |   Central NJ Meetup Part Two? Newer »

You are not logged in, either login or create an account to post comments