I already asked pb about this login problem

October 31, 2011 3:24 AM

Does anyone else have this problem with Firefox?

Yesterday, a random bug appeared, whereby if I am on www.metafilter.com, or any of the pages that are normally blue (Recent Activity, user profile, etc), I am not logged in. Here's what happens:

In Firefox 7.0.1 on Windows 7:
1) Go to www.metafilter.com (or Recent Activity, etc)
2) Discover I am not logged in. Click on login link.
3) Get to login page. Attempt to log in.
4) Find myself back at www.metafilter.com (or the page I started at), still not logged in.
5) Click on any of the subsites.
6) Find myself on a subsite page, logged in.

Steps I've taken (after consulting with pb):
1) Clear cookies.
2) Disable all browser extensions.
3) Clear cache.

Other things possibly relevant:
1) If I use Chrome, I don't have this problem.
2) Might using Firefox Sync have any bearing on this?
3) I'm in the UAE, where ISPs use major firewalls.

I am totally confused and would appreciate any suggestions that other MeFites might have about how to get the Firefox login to work again. Thanks!

posted by bardophile to Bugs at 3:24 AM (62 comments total)

Maybe the ISP is caching the main page, so instead of loading the main page showing you as logged in, it sees you're requesting www.metafilter.com again and just gives you the cached page. When you load a subsite, it fetches a fresh version because it doesn't have a cached page for you.
posted by EndsOfInvention at 3:46 AM on October 31, 2011

Why would it have the login page cached, but not the subsite?
posted by bardophile at 3:55 AM on October 31, 2011

1) Go to www.metafilter.com (or Recent Activity, etc)
2) Discover I am not logged in. Click on login link.
3) Get to login page. Attempt to log in.
4) Find myself back at www.metafilter.com (or the page I started at), still not logged in.
5) Click on any of the subsites.
6) Find myself on a subsite page, logged in.

Truly, only those pure of heart may approach the blessed realm - for it is protected by strange magicks that prevent access to goblin-kind. Galadrahowie hath cast a spell upon its door, where Elvish letters carved in mithril describe the formula: "Ask, Me, and Enter!" Thus the initiate knoweth that he must pass through the subsites of Moria, yay, and journey even unto the depths of a realm most foul, and there answer the relationship questions posed unto him, and bring not irrelevant material into such discussions, nor stray from the question as asked. Only by passing this test shall the Elf-friend emerge into the true light of MetaFilter, which is Galadrahowie's kingdom. So that's basically your problem there.
posted by the quidnunc kid at 4:45 AM on October 31, 2011 [5 favorites]

Works fine for me with FF7.0.1 and Win7. Maybe it's the redirect back from the log-in page that's causing the issue?
posted by SyntacticSugar at 5:07 AM on October 31, 2011

I use Firefox 7 on Windows 7 at home, and Firefox 8 beta on Linux at home, both Synced together. Never had this problem, although very rarely (once or twice a year maybe) I'll be suddenly logged out in the middle of a session and have to log back in. But I've never had the weird behavior you're describing here.
posted by Plutor at 5:14 AM on October 31, 2011

(Ugh, Firefox 8 beta on Linux at work.)
posted by Plutor at 5:14 AM on October 31, 2011

If it were the ISP cache, or the redirect, what might I be able to do to fix it?
posted by bardophile at 5:22 AM on October 31, 2011

In my last email I mentioned that you can rule out the redirect problem by typing in the login URL directly into your browser:

https://login.metafilter.com/

Don't click the login link, just type that URL into your browser address bar. That will take any redirects out of the equation and you can see if that's even a factor.

You didn't mention Firefox Sync before. I'm not sure how it could be causing a problem, but try disabling it and see if you still have the problem.
posted by pb (staff) at 6:39 AM on October 31, 2011

Yeah, I tried typing it directly. Didn't make any difference. I have just finished uninstalling and reinstalling Firefox. It seems to have fixed the problem, for now.

Thanks for the help, yesterday and today.
posted by bardophile at 6:47 AM on October 31, 2011

Great to hear it's working.
posted by pb (staff) at 6:51 AM on October 31, 2011

Not precisely the same issue, but I have been randomly logged out every so often for the last couple of months. Seems to coincide with upgrading to Firefox 7 (on Windows 7). I tried clearing my cache and cookies, and it seemed to get better for a while, but then returned. Hard to say if it really made a difference, since it's not easily reproducible. It happens maybe two or three times a week. I always use the same entry point: http://www.metafilter.com/home/recentcomments . I'll just be browsing through the site, and suddenly see the login link at the top of the page. I'll log in and everything is kosher again.

It's not a huge deal, except that it screws up the MetaFilter Scroll Tag greasemonkey script. If I'm not logged in, it forgets which threads I've read.
posted by team lowkey at 1:02 PM on October 31, 2011 [1 favorite]

Browsers do have limits on the amount of cookie data it will store per domain. Once you reach that limit, it starts dropping cookies. That would definitely log you out until you reset your cookies by logging again.

We don't set enough data in cookies to get anywhere close to the upper limit. However, I'm not sure how the MeFi Scroll Tag script stores data. If it's storing a large amount of data on the client side, there's the potential that it could be reaching the upper limit of what Firefox is willing to store per domain. I have no idea if that's what is happening, but it might be worth looking into.
posted by pb (staff) at 1:25 PM on October 31, 2011

I have the same problem team lowkey has. I think dg mentioned he's had it too.

I also have the issue that the site (?) forgets which threads I've read even when logged in.

I've cleared cookies, etc., but it still happens. I'm using Win7 and Firefox 7.0.1. However, it started before updating to FF7.

It's not really important, just annoying.
posted by deborah at 1:37 PM on October 31, 2011

Oh that's interesting, because I was having trouble with the Scroll Tag script a couple of days ago, and ended up disabling it.
posted by bardophile at 1:37 PM on October 31, 2011

Are you using any MetaFilter-specific Greasemonkey scripts, deborah?
posted by pb (staff) at 1:46 PM on October 31, 2011

I got a J-run error yesterday, very retro! Nothing since, and the whole 'net going overseas was a bit flaky at the time.
posted by Abiezer at 2:11 PM on October 31, 2011

Yeah, we had a server hiccup around 11:45 pm Pacific Time last night.
posted by pb (staff) at 2:13 PM on October 31, 2011

pb: "Browsers do have limits on the amount of cookie data it will store per domain. Once you reach that limit, it starts dropping cookies. That would definitely log you out until you reset your cookies by logging again."

Oh, that's interesting. I hadn't considered that. Turns out I do have a huge number of metafilter specific cookies. Looks like maybe Scroll Tag saves a separate cookie for each thread?

I can't find anything recent but this guy said in 2008:

Firefox does something strange: it seems to randomly decide which cookies to keep although the last cookie set is always kept. There doesn’t seem to be any scheme it’s following at all. The takeaway? Don’t go above the cookie limit in Firefox.

That would explain the randomness of the log-out and forgetting only some threads.
posted by team lowkey at 3:08 PM on October 31, 2011

Edro?
posted by zomg at 3:19 PM on October 31, 2011

A few years ago Firefox had a per domain limit of 50 cookies. I haven't found anything that says they've increased the limit since then. So it looks like once you reach 50 cookies for a particular domain, Firefox starts randomly dropping existing cookies.

Depending on options you choose here at MetaFilter, you could end up with 20+ cookies from the site alone. If the Scroll Tag does set a new cookie for each thread you could hit the 50 cookie limit pretty quickly.
posted by pb (staff) at 3:29 PM on October 31, 2011

I ran this cookie test, and it guessed a 150 per domain limit.

I've added network.cookie.maxPerHost = 500 in about:config . Re-ran the test, and it confirmed the 500 limit. I'll see if that clears it up.
posted by team lowkey at 3:33 PM on October 31, 2011 [1 favorite]

Browsing through the cookies, it seems like Scroll Tag may be setting two cookies per thread visited, and they don't expire, so even 150 is pretty likely.
posted by team lowkey at 3:34 PM on October 31, 2011

Or rather, they expire after 10 years.
posted by team lowkey at 3:40 PM on October 31, 2011

Yes, pb, a few:

Howls of Outrage (can't remember what this is for)
Mefiquote
MetaFilter Scroll Tag (I can probably get rid of this one since we have the yellow arrow now)
MeFi Navigator
Metafilter Unicorn and Narwhal Buttons
Mondo Image
Jessamyn's Star

I've had all of these, except the Star, before the issue started.
posted by deborah at 5:09 PM on October 31, 2011

Ah, Howls allows you to see who favourited a comment/post by floating your cursor over the link. That's fairly recent and I don't recall if it was installed before or after the issue started.

I also have the deleted posts script, but that's an oldie.
posted by deborah at 5:13 PM on October 31, 2011

Sounds like MetaFilter Scroll Tag is probably the culprit here.
posted by pb (staff) at 5:14 PM on October 31, 2011

You'll want to clear out your MeFi cookies manually in addition to uninstalling it if you go that route.
posted by pb (staff) at 5:16 PM on October 31, 2011

I've had the getting logged out problem too. Checking my cookies, there were a million or so set, two per thread, so Scroll Tag must be a Bad Thing for cookie limits. I deleted the Scroll Tag cookies, and also am trying team lowkey's suggestion for the network.cookie string, so thanks everyone for all the info!
posted by donnagirl at 7:29 PM on October 31, 2011

deborah: "Yes, pb, a few:

Howls of Outrage (can't remember what this is for)
Mefiquote
MetaFilter Scroll Tag (I can probably get rid of this one since we have the yellow arrow now)
MeFi Navigator
Metafilter Unicorn and Narwhal Buttons
Mondo Image
Jessamyn's Star

I've had all of these, except the Star, before the issue started."

I've had this problem too (rare and intermittent, so no biggie. I have the following scripts installed:
Mefiquote
HowlsOfOutrage (shows users that favourited a comment on hover)
IfIDoThisWillYouPleaseShutUp
MetaFilter Scroll Tag
MeFi Navigator

It makes sense that Scroll Tag is the culprit - I'll try disabling it and see what happens. Intermittent problem at worst, so could take a while to notice any results.
posted by dg at 10:34 PM on October 31, 2011

What dg said. Thanks for the help, pb.
posted by deborah at 11:06 PM on October 31, 2011

donnagirl: "Checking my cookies, there were a million or so set, two per thread, so Scroll Tag must be a Bad Thing for cookie limits"

Well, shit. Sorry everyone. I should make these not last 10 years, eh?
posted by Plutor at 10:26 AM on November 3, 2011 [1 favorite]

Crud. My work-around seemed to be working (I hadn't been randomly logged out or lost count of a thread), but then I was rummaging through the archives, (which adds a bunch of Scroll Tag cookies), and ended getting a "400 Bad Request" error on every *.metafilter.com page. I'm not positive it's related, but I had to clear my cookies to get back on the site.

400 Bad Request

Your browser sent a request that this server could not understand.

Size of a request header field exceeds server limit.
Cookie: __utmz=1.1320813961.55.6.utmcsr=metatalk.metafilter.com|utmccn=(referral)|utmcmd=referral|utmcct=/21175/Gotta-say-it-was-a-good-day; mst_metatalk.21099=936228; mst_metatalk.21099num=39; mst_metatalk.21102=938025; mst_metatalk.21102num=113; mst_metatalk.21094num=1; mst_metatalk.21103=937103; mst_metatalk.21103num=25; mst_www.107650=3933910; mst_www.107650num=187; mst_metatalk.21111=937147; mst_metatalk.21111num=22; mst_www.108236=3989478; mst_www.108236num=50; mst_www.11744=161342; mst_www.11744num=61; mst_www.108677=3989905; mst_www.108677num=89; mst_metatalk.21114=937237; mst_metatalk.21114num=1; mst_metatalk.21115=937283; mst_metatalk.21115num=11; mst_www.108713=3991471; mst_www.108713num=51; mst_www.108715=3991645; mst_www.108715num=23; mst_www.108697=3991668; mst_www.108697num=63; mst_www.108683=3991783; mst_www.108683num=100; mst_metatalk.21113=937344; mst_metatalk.21113num=19; mst_metatalk.21108=942456; mst_metatalk.21108num=148; mst_www.108720=3992276; mst_www.108720num=34; mst_www.108724=3992711; mst_www.108724num=87; mst_www.108730=3992721; mst_www.108730num=18; mst_www.108755=3994718; mst_www.108755num=124; mst_www.108759=3994608; mst_www.108759num=45; mst_www.108742=3993920; mst_www.108742num=62; USER_ID=20074; USER_NAME=team%20lowkey; USER_TOKEN=59FC6E9CED6824D002012B7E0E43F413; USER_KEY=CD0B7A0D3C64B3A8253CFEF36E5B2586; FONT_SIZE=12; SMALL_FONT_SIZE=10; TIMEZONE=0%2E0; TITLES=1; THEME=1; mst_metatalk.21123=937773; mst_metatalk.21123num=1; mst_www.108765=3994294; mst_www.108765num=28; mst_www.108739=3994049; mst_www.108739num=17; mst_www.108769=3994231; mst_www.108769num=17; mst_www.108770=3994262; mst_www.108770num=30; mst_www.108767=3994282; mst_www.108767num=13; mst_www.108772=3994315; mst_www.108772num=19; mst_www.108771=3994199; mst_www.108771num=15; mst_www.108779=3994563; mst_www.108779num=1; mst_www.105556=3815313; mst_www.105556num=58; mst_metatalk.21125=938042; mst_metatalk.21125num=99; mst_metatalk.21121=937733; mst_metatalk.21121num=55; mst_www.108783=3994830; mst_www.108783num=27; mst_www.108786=3994857; mst_www.108786num=15; mst_www.108788=3994866; mst_www.108788num=1; mst_www.108789=3994876; mst_www.108789num=4; mst_www.108800=3996203; mst_www.108800num=297; mst_www.108796=4002333; mst_www.108796num=336; mst_www.108814=3996465; mst_www.108814num=32; mst_www.108811=3996029; mst_www.108811num=45; mst_www.108824=3996836; mst_www.108824num=10; mst_www.108799=3995293; mst_www.108799num=9; mst_metatalk.21129=938818; mst_metatalk.21129num=73; mst_metatalk.21120=937601; mst_metatalk.21120num=14; mst_www.93534=3175159; mst_www.93534num=271; mst_www.108867=3998921; mst_www.108867num=76; mst_www.108834=3998939; mst_www.108834num=28; mst_metatalk.21132=939214; mst_metatalk.21132num=113; mst_www.108870=4002655; mst_www.108870num=320; mst_www.108893=4000235; mst_www.108893num=56; mst_www.108902=4000729; mst_www.108902num=142; mst_metatalk.21134=939305; mst_metatalk.21134num=1; mst_metatalk.21136=939368; mst_metatalk.21136num=12; mst_www.108910=4000818; mst_www.108910num=16; mst_www.108916=4001157; mst_www.108916num=12; mst_metatalk.21138=939483; mst_metatalk.21138num=31; mst_www.108930=4001919; mst_www.108930num=2; mst_metatalk.21139=939898; mst_metatalk.21139num=180; __utma=1.1949447615.1319998900.1320819167.1320821942.57; __utmv=1.|1=User%20Type=member=1; mst_www.85779=2777185; mst_www.85779num=23; mst_metatalk.21130=940532; mst_metatalk.21130num=863; mst_metatalk.21144=940116; mst_metatalk.21144num=5; mst_metatalk.21145=941421; mst_metatalk.21145num=31; mst_www.108994=4010905; mst_www.108994num=211; mst_www.108981=4003603; mst_www.108981num=17; CFID=1174407407; CFTOKEN=258842fe976ab4d6-87574192-19B9-C893-9BD9107B5A0FE2C0; mst_metatalk.21143=939994; mst_metatalk.21143num=34; mst_metatalk.21147=942428; mst_metatalk.21147num=94; mst_www.108995=4004208; mst_www.108995num=9; mst_www.108945=4005116; mst_www.108945num=26; mst_metatalk.21148=941253; mst_metatalk.21148num=57; mst_www.108982=4005370; mst_www.108982num=50; mst_www.109008=4004861; mst_www.109008num=1; mst_www.109002=4004653; mst_www.109002num=1; mst_ask.47496=723444; mst_ask.47496num=22; mst_www.109003=4005381; mst_www.109003num=37; mst_www.100937=3543505; mst_www.100937num=22; mst_metatalk.21146=940223; mst_metatalk.21146num=4; mst_ask.199799=2876159; mst_ask.199799num=1; mst_ask.199738=2875591; mst_ask.199738num=14; mst_metatalk.21149=941497; mst_metatalk.21149num=186; mst_www.81300=2548348; mst_www.81300num=5; mst_www.109012=4005127; mst_www.109012num=1; mst_www.108854=3998079; mst_www.108854num=1; mst_metatalk.21150=941047; mst_metatalk.21150num=74; mst_www.109033=4006037; mst_www.109033num=19; mst_metatalk.20989=935359; mst_metatalk.20989num=1779; mst_metatalk.21152=940884; mst_metatalk.21152num=8; mst_www.109034=4006228; mst_www.109034num=30; mst_www.109023=4005612; mst_www.109023num=8; mst_metatalk.21153=940889; mst_metatalk.21153num=3; mst_www.109020=4005534; mst_www.109020num=20; mst_metatalk.21151=941653; mst_metatalk.21151num=85; mst_metatalk.21155=941237; mst_metatalk.21155num=77; mst_metatalk.19949=829430; mst_metatalk.19949num=78; CONTACTVIEW=list; mst_www.109057=4007769; mst_www.109057num=139; mst_www.109053=4007239; mst_www.109053num=42; mst_metatalk.21157=941493; mst_metatalk.21157num=30; mst_metatalk.21156=941259; mst_metatalk.21156num=1; mst_www.109065=4008563; mst_www.109065num=53; mst_www.109072=4010978; mst_www.109072num=337; mst_metatalk.19050=755323; mst_metatalk.19050num=128; mst_metatalk.21158=941517; mst_metatalk.21158num=48; mst_www.108887=4000000; mst_www.108887num=31; mst_www.109079=4009229; mst_www.109079num=31; mst_www.109092=4010171; mst_www.109092num=6; mst_www.109082=4010188; mst_www.109082num=60; mst_www.109093=4010469; mst_www.109093num=113; mst_www.109086=4010173; mst_www.109086num=51; mst_www.109089=4010369; mst_www.109089num=66; mst_www.109084=4009545; mst_www.109084num=63; mst_www.109101=4010982; mst_www.109101num=46; mst_www.109112=4011404; mst_www.109112num=70; mst_www.109124=4012213; mst_www.109124num=81; mst_metatalk.21159=941604; mst_metatalk.21159num=77; mst_www.109104=4011009; mst_www.109104num=18; mst_www.109096=4012610; mst_www.109096num=89; mst_metatalk.21161=941685; mst_metatalk.21161num=26; mst_metatalk.21164=941853; mst_metatalk.21164num=8; mst_www.109133=4012579; mst_www.109133num=33; mst_metatalk.21163=941772; mst_metatalk.21163num=4; mst_metatalk.21165=941996; mst_metatalk.21165num=1; mst_www.109168=4014449; mst_www.109168num=68; mst_www.109170=4014339; mst_www.109170num=7; mst_metatalk.21166=943009; mst_metatalk.21166num=328; mst_www.91634=3072715; mst_www.91634num=1; mst_www.109167=4014236; mst_www.109167num=13; mst_metatalk.21170=942534; mst_metatalk.21170num=85; mst_metatalk.21168=942368; mst_metatalk.21168num=3; mst_metatalk.21169=942373; mst_metatalk.21169num=1; mst_www.108732=3992584; mst_www.108732num=1; mst_www.109178=4017072; mst_www.109178num=38; __utmc=1; mst_ask.34705=540723; mst_ask.34705num=16; mst_metatalk.21174=942711; mst_metatalk.21174num=22; mst_metatalk.21172=943012; mst_metatalk.21172num=196; mst_metatalk.21175=943021; mst_metatalk.21175num=33; mst_ask.200460=2886842; mst_ask.200460num=74; mst_metatalk.21173=943044; mst_metatalk.21173num=97; mst_www.104017=3725747; mst_www.104017num=60; mst_www.103863=3718721; mst_www.103863num=39; mst_metatalk.18253=686808; mst_metatalk.18253num=188; mst_ask.31713=497708; mst_ask.31713num=1; mst_www.88557=2916514; mst_www.88557num=113; mst_www.85624=2768968; mst_www.85624num=128; mst_www.77175=2366565; mst_www.77175num=24; mst_www.84668=2722138; mst_www.84668num=144; mst_metatalk.17825=654023; mst_metatalk.17825num=166; mst_metatalk.14375=423479; mst_metatalk.14375num=80; mst_metatalk.17671=641874; mst_metatalk.17671num=84; mst_metatalk.17410=619851; mst_metatalk.17410num=127; mst_metatalk.17391=619557; mst_metatalk.17391num=61; mst_metatalk.17366=616709; mst_metatalk.17366num=408;
posted by team lowkey at 10:01 AM on November 9, 2011

Yeah, definitely related team lowkey. All of those mst_* cookies are from the MetaFilter Scroll Tag script. All of your cookies are sent to us on every request and our server hit its limit of what it was willing to accept.

Sending that much data on each request could impact how fast the site loads for you too. Maybe Plutor could switch to using a different storage mechanism. Greasemonkey has a local database if I'm remembering right. Modern browsers also have HTML5 Web Storage which GM might be able to tap into, I'm not sure.
posted by pb (staff) at 10:08 AM on November 9, 2011

Looks like he tried to go down that road, and had to go back to cookies. What would be really awesome is if Firefox just deleted the oldest cookie when it hit its limit, instead of this random nonsense.
posted by team lowkey at 11:26 AM on November 9, 2011

I added this to my copy of MetaFilter Scroll Tag:

var cookie_limit = 100;

//
// mst_limit_cookies
//
// If number of cookies exceeds cookie_limit, expire the two oldest cookies.
//

function mst_limit_cookies(limit) {
    var ca = document.cookie.split(';');       //get array of cookies for current page
    var mst_ca = new Array();                  //new array for mst cookies
    for(var i=0;i < ca.length;i++) {           //traverse cookie array
        var cookie = ca[i];                                            
        if (cookie.search("mst_") == 1) {      //if current cookie starts with mst_
           mst_ca.push(cookie.split('=')[0]);  //put it into the mst cookie array
        }
    }
    if (mst_ca.length >= limit) {                               //if mst cookie array above limit
        Cookie.unset(mst_ca[0], undefined, "metafilter.com");   //expire first cookie
        Cookie.unset(mst_ca[1], undefined, "metafilter.com");   //expire second cookie
    }
}

And call the function at the top of mst_initThread:

function mst_initThread() {
    mst_limit_cookies(cookie_limit);
...

I've never written javascript before, so I can't guarantee anything, but what it should do is check if there are 100 mst cookies or more, and expire the two oldest (if in fact document.cookie returns the oldest cookies first, which it appears to).

What that means is Scroll Tag should only keep track of the latest 50 threads visited (two cookies per thread), so it won't go over the browser's cookie limit. I'll try it for a while and see if it breaks anything.
posted by team lowkey at 9:22 PM on November 9, 2011

And I just realized, I should delete cookies until the limit is reached. Corrected:

function mst_limit_cookies(limit) {
    var ca = document.cookie.split(';');       //get array of cookies for current page
    var mst_ca = new Array();                  //new array for mst cookies
    for(var i=0;i < ca.length;i++) {           //traverse cookie array
        if (ca[i].search("mst_") == 1) {       //if current cookie starts with mst_
           mst_ca.push(ca[i].split('=')[0]);   //put it into the mst cookie array
        }
    }
    if (mst_ca.length >= limit) {                        //if mst cookie array above limit
        for (var i=0;i <= mst_ca.length - limit;i++) {   //expire every cookie up to limit
            Cookie.unset(mst_ca[i], undefined, "metafilter.com");
        }
    }
}

posted by team lowkey at 10:41 PM on November 9, 2011

Just to follow up (if anyone cares). The script changes have been working. No random log-outs or forgotten threads since I implemented it. I did have to adjust the cookie limits, only because I was hitting the 50 thread limit way too quickly.

Metafilter seems to refuse the request at around 2150 characters, which happened to be around 320 various sized cookies.

In about:config, I set:

network.cookie.maxPerHost = 250

And then in the Scroll Tag script I upped:

var cookie_limit = 220;

So Scroll Tag remembers 110 threads instead of 50, leaving 30 cookies for other purposes. I wish it could remember everything forever, but that can't happen without changing the storage mechanism, which is way out of my league.
posted by team lowkey at 2:02 PM on November 23, 2011

I care!

After disabling Scroll Tag the problem has ceased to exist which is good as far as it goes. I really miss the ability to click it and be sent down to the last comment I read.
posted by deborah at 6:36 PM on November 24, 2011

Care factor >0 here, too. Disabling MeFi Scroll Tag has stopped the random log-outs for me as well, but I really miss the functionality it provides, so I'm debating whether to re-enable it and deal with the log-outs. I can't decide which is more annoying.
posted by dg at 6:56 PM on November 24, 2011

You can get my altered Scroll Tag script here, if you'd like:

http://lowkey.org/metafilter_scroll_tag.user.js

But you also need to change your Firefox configuration. Type about:config in the title bar, right-click a preference, and then choose New->Integer.

Enter the preference name "network.cookie.maxPerHost" and give it the value of "250". That will get you a Scroll Tag that remembers the last 110 posts you've visited, with no random log-outs.

And to be clear, this is Plutor's script with an added function (which is not optimized) that deletes older cookies. I bet Plutor will come up with a better method when he has time for it. This is just a tribute.
posted by team lowkey at 2:09 AM on November 25, 2011 [1 favorite]

I care, too! Thanks for the new script, team lowkey. I actually was able to stop the logout problem by following just your original suggestion about the network.cookie.maxPerHost value. It introduced a different weird error, where every now and then mefi would tell me I'd sent a bad request, but it happened way less frequently than the logout thing and deleting cookies solved it immediately. I'm looking forward to your modified script handling deletions for me, thanks!
posted by donnagirl at 7:38 PM on November 25, 2011

Yeah, that bad request thing is because the browser sends ALL THE COOKIES! every time you load a page, and that gets too big for MeFi's server to handle. Bringing down the max number of cookies to around 250 fixes that. The random log-out will happen less often because there are more cookies to choose from when it does a random delete, so it's less likely to delete the cookie that keeps you logged in. I didn't know about any of that until pb pointed it out.

This is my first time hacking on Greasemonkey, and there are some obvious ways to improve this cookie deletion function, if anyone out there has more experience and can help out.

The first thing would be to fetch the Firefox preference for max cookies and expire cookies based on that limit instead of defining the limit in the script itself. I tried this get preferences method, but I couldn't get it to work.

The second thing is to find and remove the oldest mst cookies in the existing set, instead of creating a new array and working with that. It's not a huge deal, but it's not optimal to build a new array. I did it to count the mst_ cookies, since I couldn't figure out how to get the browser's global limit. And because it was easy as a first attempt.

The best thing would be to store the data without using cookies. I don't know how to do that at all, and it sounds like Plutor tried and couldn't get it to work. I'm guessing that's why we didn't see this problem until a few months ago, when he switched to cookies with a ten year expiration for storage.
posted by team lowkey at 12:35 AM on November 26, 2011

Hey sorry I haven't been following this so closely, and that I've totally ignored this edge case. (You're edge cases, I say!)

I like your fix, team lowkey, but I just want to call it from a different place -- when writing a cookie instead of every time a thread is loaded. Plus you make no attempt to see what you're deleting. Your code keeps the "first" 220 cookies, where first is defined by the order that document.cookie hands them to you. I'm not sure that order is dependable.

team lowkey: "The best thing would be to store the data without using cookies. I don't know how to do that at all, and it sounds like Plutor tried and couldn't get it to work."

The problem is that Scroll Tag is a much-beloved script, and it therefore is the script I get the most questions about cross-browser support. DOM Storage is supported by Opera and Chrome and Firefox, but they all support it very slightly differently, and I had trouble getting it to work dependably across the board. For the functionality for the "Recent Activity" page to work, it needs to be able to see this data across-subdomains, which isn't something DOM Storage does (what the eff, W3C?).

Given the velocity of browser development (especially Chrome and Firefox) maybe it's time to give it another try. I wish there was a tiny drop-in library I could use that would do the right thing.
posted by Plutor at 5:33 AM on November 28, 2011

Wait, why am I using a billion cookies for this data? It's not a lot of info, I could pack it pretty easily into one cookie (or a handful -- like one for each subsite).
posted by Plutor at 5:39 AM on November 28, 2011

Plutor: "I like your fix, team lowkey, but I just want to call it from a different place -- when writing a cookie instead of every time a thread is loaded."

Oh. Yeah, you're totally right. I didn't really look closely, and assumed mst_initThread was only called when visiting a new thread, and thus when it would be writing a new cookie. My bad.

Plutor: "Your code keeps the "first" 220 cookies, where first is defined by the order that document.cookie hands them to you. I'm not sure that order is dependable."

Yeah, I'm not sure either. I was initially going to sort the cookies, but it seemed to return them in historical order, so I went with it. Because I'm lazy.

Plutor: "Wait, why am I using a billion cookies for this data? It's not a lot of info, I could pack it pretty easily into one cookie (or a handful -- like one for each subsite)."

That would be awesome. Thanks for coming back to help out us "edge cases" ;)
posted by team lowkey at 1:20 PM on November 28, 2011

Here's a beta of my fix. Here's a quick description of my new cookie storage (haven't had the time to re-investigate DOM Storage yet):

* Each subsite has one cookie
* For each thread, I need to store three pieces of data: thread_id, comment_id, and the number of comments read
* To balance maximum density and conversion simplicity, I'm storing those three numbers in base 64 (approximately uuencoded)
* I'm using fixed-length values instead of separators. thread_id and comment_id get 4 characters each, and comments read gets 2. So we'll be good up through comment number 16,777,216, which is about quadruple where we are right now. (Unfortunately, threads with more than 4096 comments will start to behave a little erratically, but those are rare enough that I'm okay with this)
* Pretty much every browser maxes cookies out around 4096 bytes. Since we pack each thread into 10 bytes, we can store just over 400 threads per subsite.

This version of the scroll tag falls back to the "old" cookie method and re-stores those values if it finds them. Give this a whirl for a couple of days. When we're all happy enough with it, I'll make it also delete the old style cookies it finds (or you can do so manually).
posted by Plutor at 5:43 AM on November 29, 2011 [1 favorite]

Thanks, Plutor.

I installed your new version a few hours ago, and it just now "forgot" a few of the threads I visited today. I was already pushing the browser's max cookie limit, so it's probably just the browser cleaning up, but I thought I'd mention it.

I've now manually removed all of the old mst_ cookies and will start fresh. I'll let you know how it goes.
posted by team lowkey at 2:35 PM on November 29, 2011

Urg... something's definitely hinky. I read to the bottom of this thread, went back to Metatalk, and it said there were 47 comments (42 new). Clicking the 42 new brought me down to the bottom of the thread, though.

Went back to Metatalk, and it showed that there were no new comments on this thread, but had forgotten all the other threads I had visited. I read to the bottom of another thread, went back to MetaTalk, and it showed no new comments for that thread, but had forgotten this one again.

The content of the current mst_metatalk cookie is: AFKZDnucAvAFLyDnt1Jb
And the expiration is: Friday, November 26, 2021 3:01:30 PM

So I guess it overwrote the old cookie with a new one? If I'm understanding your storage method correctly, that string could only contain info about two threads, right?
posted by team lowkey at 3:17 PM on November 29, 2011

Welcome back, old friend! Thanks Plutor - I really missed the functionality that the script provided and it's great to have it back.
posted by dg at 3:18 PM on November 29, 2011

Yeah, it's definitively overwriting the data for some reason. I went and clicked through a couple more threads, and it forgot everything again. Current mst_metatalk cookie is now:
content: AFLxDnfVAW
expiration: Friday, November 26, 2021 3:19:26 PM.
posted by team lowkey at 3:24 PM on November 29, 2011

Hm. I just tried to do what you described, and it seems to be working okay for me. Do you have Firebug installed? If so, can you open it, go to the console, select "Persist" and "All", and then do your two or three steps, and then copy and paste all of the console content into a MeMail or email? I kept a lot of debugging on in that beta for exactly this reason.

(Also, what versions of Firefox and Greasemonkey?)
posted by Plutor at 3:35 PM on November 29, 2011

Firefox 8.0.1, Greasemonkey 0.9.13

I installed Firebug, but copying data out of the console isn't formatted very well. Here's what seems to be a useful snippet, while I'm scrolling down a page. I turned on the "data before" debug message, and added debug info about the trim function, since that seems to be called every time and would be a likely candidate for removing data:

getValue metatalk 21243
Dnuf A
949151 0
setValue metatalk 21243 949143 1
AFL7 DnuX AB
data before: AFL6DntOAOAFL5DntrAIAFL7DnufA
cdata.length: 29 cookie_limit: 4096 floor: 4090
data after: AFL7DnuXABAFL6DntOAOAFL5DntrA
setValue metatalk 21243 949144 2
AFL7 DnuY AC
data before: AFL7DnuXABAFL6DntOAOAFL5DntrA
cdata.length: 29 cookie_limit: 4096 floor: 4090
data after: AFL7DnuYACAFL6DntOAOAFL5DntrA

So that shows that the data changed pretty significantly, and that it's length isn't a multiple of 10. I'm suspecting that maybe the translation to base 64 is setting "0" to "A", which would muck up the string length, and all subsequent transformations, right?
posted by team lowkey at 4:53 PM on November 29, 2011

And I might very well be barking up the wring tree. I enabled more debugging messages and am emailing the output to you, Plutor.
posted by team lowkey at 5:06 PM on November 29, 2011

Thanks for the email and the extra logging. You're right about what's happening, you're just wrong about where. It keeps the cookie sorted in chronological order -- most recently visited threads first. So it's got to extract that thread from the middle of the string first. And of course that code has an off-by-one mistake.

The location posted above is updated with a two-character fix.
posted by Plutor at 5:28 PM on November 29, 2011

Thanks.

I just updated to the new version, cleared my old cookie, and it seemed to be doing fine, but then as I was scrolling down another thread, it lost data again:

setValue metatalk 21227 947582 36
AFLr DnV+ Ak
data after: AFLrDnV+AkAFL7DnuiAIAFL8Dnv/Ar
setValue metatalk 21227 947583 37
AFLr DnV/ Al
data after: AFLrDnV/Al

Multiples of ten now, but something's still breaking. I'll turn the debugging messages back on and try to pin it down.
posted by team lowkey at 8:20 PM on November 29, 2011

Again, with debugging on. Different thread, same result.

setValue metatalk 21242 949054 3
AFL6 Dns+ AD
data before: AFL6Dns9ACAFL7DnwPA1AFL8Dnu5ABAFKZDnvSA2AFLrDnWVA4
data 1: AFL7DnwPA1AFL8Dnu5ABAFKZDnvSA2AFLrDnWVA4
data 2: AFL6Dns+ADAFL7DnwPA1AFL8Dnu5ABAFKZDnvSA2AFLrDnWVA4
cdata.length: 50 cookie_limit: 4096 floor: 4090
data after: AFL6Dns+ADAFL7DnwPA1AFL8Dnu5ABAFKZDnvSA2AFLrDnWVA4
setValue metatalk 21242 949055 4
AFL6 Dns/ AE
data before: null
cdata.length: 10 cookie_limit: 4096 floor: 4090
data after: AFL6Dns/AE

Cookie.get("mst_" + subsite) is coming up blank. It's interesting that it happened again after a "+" was put in the cookie. I'll run through a couple more times and see if that matters.
posted by team lowkey at 8:38 PM on November 29, 2011

Yeah, twice more now. After a "+" gets entered into the cookie, it returns empty.

setValue metatalk 21244 949182 3
AFL8 Dnu+ AD
data before: AFL8Dnu7ACAFKZDnwQA3AFL6DntBAF
data 1: AFKZDnwQA3AFL6DntBAF
data 2: AFL8Dnu+ADAFKZDnwQA3AFL6DntBAF
cdata.length: 30 cookie_limit: 4096 floor: 4090
data after: AFL8Dnu+ADAFKZDnwQA3AFL6DntBAF
setValue metatalk 21244 949183 4
AFL8 Dnu/ AE
data before: null
cdata.length: 10 cookie_limit: 4096 floor: 4090
data after: AFL8Dnu/AE

...

setValue metatalk 21244 949246 42
AFL8 Dnv+ Aq
data before: AFL8Dnv9ApAFKZDnwZA4
data 1: AFKZDnwZA4
data 2: AFL8Dnv+AqAFKZDnwZA4
cdata.length: 20 cookie_limit: 4096 floor: 4090
data after: AFL8Dnv+AqAFKZDnwZA4
setValue metatalk 21244 949247 43
AFL8 Dnv/ Ar
data before: null
cdata.length: 10 cookie_limit: 4096 floor: 4090
data after: AFL8Dnv/Ar
posted by team lowkey at 8:45 PM on November 29, 2011

Word. I took "+" out of the base64chars, and everything's working fine (don't know why that character matters, but there you go). Gonna hafta either replace that character with something else, or reduce the limit accordingly.
posted by team lowkey at 8:51 PM on November 29, 2011

Probably something like "+" is special in the RegExp in document.cookie.match .
posted by team lowkey at 8:59 PM on November 29, 2011

Nice bug squashing, guys. It's not so much that + is special in RegExps as it is that the cookie-js code I stole am using doesn't consider it a "valid" cookie character. No big loss. I've changed the + in my base 64 alphabet to an underscore. Updated beta available at the same place.
posted by Plutor at 6:29 AM on November 30, 2011

It forgot visited threads again, though it took a while.

I didn't catch the output this time, since debugging was turned off. I'll see if I can reproduce it.

My current cookie length is 281, so it looks like it's another offset issue, not losing the cookie entirely. Here's the cookie:

AFKZDnxKA8AFMADnyxAfAFMBDnyyAMAFLPDnxmAAFLyDnxvJgAFL6Dnx/AyAFL7Dnx_BfAFLzDntiBlAFMADnyaAaAFLrDnwMDxAFL5DnySAuAFLLDnxfCXAFL8DnxuBJAFL_DnygASAFKZDnxKA8AFL/DnydA_Dnx6ARAFKZDnxKA8AFMADnyaA8Dnw0A5AFL6DnwwBbAFKdDnYaC8AFL/DnwvAFAFLzDntiBlAFLyDnwjJdAFLLDnuWCVAFLxDnuuBbAFLrDnwMDxAFL5DnwWAq
posted by team lowkey at 1:02 PM on November 30, 2011

From the end of one thread:

setValue metatalk 21233 949438 93
AFLx Dny_ Bd
data before: AFLxDny0BcAFL_DnzAAUAFMBDnzEAXAFL/DnzGAFAFLrDnwMDxAFKZDnzJA9AFLyDnxvJgAFMADnzKAlAFL8DnzLBLAFMCDnzMAB
data 1: AFL_DnzAAUAFMBDnzEAXAFL/DnzGAFAFLrDnwMDxAFKZDnzJA9AFLyDnxvJgAFMADnzKAlAFL8DnzLBLAFMCDnzMAB
data 2: AFLxDny_BdAFL_DnzAAUAFMBDnzEAXAFL/DnzGAFAFLrDnwMDxAFKZDnzJA9AFLyDnxvJgAFMADnzKAlAFL8DnzLBLAFMCDnzMAB
cdata.length: 100

And then going back to a previously seen thread:

setValue metatalk 21250 949452 1
AFMC DnzM AB
data before: AFLxDny_BdAFL_DnzAAUAFMBDnzEAXAFL/DnzGAFAFLrDnwMDxAFKZDnzJA9AFLyDnxvJgAFMADnzKAlAFL8DnzLBLAFMCDnzMAB
data 1: AFLxDny_BdAFL_DnzAAUAFMBDnzEAXAFL/DnzGAFAFLrDnwMDxAFKZDnzJA9AFLyDnxvJgAFMADnzKAlAFL8DnzLB
data 2: AFMCDnzMABAFLxDny_BdAFL_DnzAAUAFMBDnzEAXAFL/DnzGAFAFLrDnwMDxAFKZDnzJA9AFLyDnxvJgAFMADnzKAlAFL8DnzLB
cdata.length: 99

Looks like it removed an extra character from the previous thread's data when removing the current thread's data from the end of the cookie. The end of "data 1:" should be AFL8DnzLBL
posted by team lowkey at 1:37 PM on November 30, 2011