pyMeFi

July 15, 2008 3:45 PM

I'm playing with a simple python MeFi text-based RSS checker. Code here.

It's simple little python script that checks the MeFi rss feed every 60 seconds, and displays any new posts.

You can keep it running in a background terminal/cmd window for a realtime, text-only mefi fix.

Requires feedparser.

Corrections/improvements welcome. There's some weirdness with the UTF encoding, for instance.

posted by signal to MetaFilter-Related at 3:45 PM (24 comments total)

Wow, that sounds weird. Can you take a screenshot of your terminal window with some output on it?
posted by mathowie (staff) at 3:46 PM on July 15, 2008

ScreenCap
posted by signal at 3:49 PM on July 15, 2008

Dude, it's like looking directly at the matrix.
posted by mathowie (staff) at 3:53 PM on July 15, 2008 [1 favorite]

That's cool. I would never use it, since I don't actually understand the following terms:

python, RSS, terminal/cmd, feedparser, UTF encoding

But the screen cap looks cool and reminds me of using lynx, pine and nn, which I barely understood but used anyway because what other option did I have? AOL?

Oh, yeah, I did that too until I got TOSsed. Don't tell anyone.
posted by dersins at 4:10 PM on July 15, 2008 [1 favorite]

Ha. That's pretty cool.
posted by eyeballkid at 4:25 PM on July 15, 2008

Signal has created something more professional looking than the plain theme. Isn't that one of things that need to happen before the apocalypse?
posted by Memo at 4:50 PM on July 15, 2008

Cool. I'm going to tinker with that tomorrow. I'm suggesting that arrows up/down select a post (roll post titles at the last line) and arrow right goes to post's RSS and arrow left back to main listing. Keyboard shortcuts to switch from (m) main into (t) metatalk and (a) ask...

I usually need to keep 3-6 terminal windows open at work. Nice addition to usual confusion.
posted by Free word order! at 6:23 PM on July 15, 2008

Free word order!: yeah, I was thinking of a somewhat simpler system where you could type in the id number of each post (the script's number, not the actual MeFi number). However, I got stuck on a simple, cross-platform, non-blocking way to read key presses in the console.
posted by signal at 6:40 PM on July 15, 2008

Forgot to add: type in the id number of each post to open it your browser.
posted by signal at 6:42 PM on July 15, 2008

60 seconds is a pretty high polling frequency. I don't expect that this will be used too widely, but if everyone polled every 60s, it'd be a bandwidth nightmare.
posted by chrisamiller at 7:09 PM on July 15, 2008

Wow, that's cool -- but every 60 seconds is quite a load on the MeFi server, no? It's generally not considered kosher to be hitting an RSS feed more than once every 60 minutes; every 15 or 30, I could maybe understand, but every single minute is quite a huge load.
posted by delfuego at 7:15 PM on July 15, 2008

Changed the timeout to 600 seconds, though I honestly doubt there will ever be more than 2 or 3 people using this at the same time (if that many).
posted by signal at 7:29 PM on July 15, 2008

The usual way to alleviate bandwidth/load concerns with RSS is to make sure reader and server both support HTTP's Conditional Get. It's really a must for RSS.
posted by sdodd at 7:36 PM on July 15, 2008

That's cool. I would never use it, since I don't actually understand the following terms:
python, RSS, terminal/cmd, feedparser, UTF encoding

dersins,
What parts of Web 0.0, Web 1.0, and Web 2.0 don't you understand?
Me neither.
posted by lukemeister at 7:47 PM on July 15, 2008

I did a little noodling around, and it doesn't appear as though the response to an RSS request (for example, this page) includes an ETag. Too bad. It does include the "Date:" header. Alas, I haven't gotten the server to return a "304 Not Modified" when I send a If-Modified-Since header with that date. So it's possible the MeFi servers don't support Conditional Get.
posted by sdodd at 7:48 PM on July 15, 2008

Matt, here's the deal:

I have modified this script to check every 5 seconds and loaded it on my 175,000-strong botnet.

I want the 60,000 * $5 in small, unmarked bills in a cool, expensive canvas laptop bag left at a location to be announced.

Seriously, this is pretty cool. I had no trouble using it on Ubuntu Hardy after I installed the python feedparser package.
posted by SteveTheRed at 8:11 PM on July 15, 2008

Posted an improved version: up/down arrows browse titles (by using unixy support for command history), enter selects them and jumps to post's comment rss and it understands these commands:
mf>
'm' (+enter) : jump to MetaFilter main page
't' : jump to MetaTalk
'a' : jump to AskMeFi
'q' : quit
Not recognized commands display help.

I'm not sure if python understands windows command line history. If that doesn't work, return to signals latest version. Also there is still unicode problem: ask doesn't work because of that.

Since it waits for user input, it doesn't update automatically, but all actions cause update.
posted by Free word order! at 8:21 PM on July 15, 2008

Fwo!: tried it, get an eror (on windows):

Traceback (most recent call last):
  File "C:\scripts\getMeFi2.py", line 23, in 
    readline.add_history(e.title)
  File "C:\Python25\Lib\site-packages\pyreadline\rlmain.py", line 161, in add_history
    self._history.add_history(line)
  File "C:\Python25\Lib\site-packages\pyreadline\lineeditor\history.py", line 88, in add_history
    if not line.get_line_text():
AttributeError: 'unicode' object has no attribute 'get_line_text'

posted by signal at 8:36 PM on July 15, 2008

It seems that pyreadline and readline behave differently here. For some reason add_history in pyreadline excepts LineObjects and in readline strings or unicode strings. I don't want to overwrite my readline, so I can't test it but if you add


from pyreadline.lineeditor.lineobj import TextLine

( I'm not sure about this import, try it in console and leave lineobj out if it doesn't work)

and change

readline.add_history(e.title) -to->
readline.add_history(TextLine(e.title))


Then this could work in windows/pyreadline, but not without pyreadline. If you can get it work, then adding conditions for which to use will be easy. I hope this works, because (crude) command line interface for mefi is a keeper.
posted by Free word order! at 2:39 AM on July 16, 2008

Yep. that fixes it.
This change makes it open the link in a browser.
posted by signal at 7:31 AM on July 16, 2008

Good, with this the code works in both windows and mac/unix/linux.
posted by Free word order! at 8:00 AM on July 16, 2008

sdodd, Conditional Get is only part of the equation -- because even with it enabled, if there's been ANY change to an RSS feed, you still get the ENTIRE RSS feed. Even if only one additional item has been added.

So yep, it definitely is a great thing for a seldom-updated site that doesn't want its users to download the entire feed 24 times a day when there's only one update that day. But it's not much of a thing at all for a site like MetaFilter that updates many, many times a day, because chances are far more than even that between polls, the content will have changed... and that change will then lead to serving up the entire RSS feed again.
posted by delfuego at 10:01 AM on July 16, 2008

Yeah, delfuego, absolutely -- it's a piece of the puzzle. ATOM encourages you to send only the most recent dozen (or so) feed entries and a link the reader can follow to fetch the next "page" worth of results. Some feed servers (like Flickr's public photo feed) send only what's new since the last time you asked. Combined with pagination, you really cut down on wasted bandwidth.

Also, the feed for this particular MeTa post demonstrates that not all of MeFi's feeds change too quickly to benefit from Conditional Get.

So if you really wanted to poll the feed every 60 seconds, you could do it without being too concerned about bandwidth. You just have to get the nitty-gritty details of RSS right.
posted by sdodd at 10:42 AM on July 16, 2008

I just want to pop in here and say "neat".

Neat.
posted by cortex (staff) at 1:56 PM on July 21, 2008