Is there some trick to making "X new" work?

July 12, 2022 11:05 PM

It will often happen that a post will say "N comments (X new)" for me, when "X" is wildly inaccurate. Sometimes there are 0 new when it tells me there are some, and other times there are some new but not as many as it says. Is there something within my control to improve how well this works?

I guess on some level, I'm asking how it works, and what causes the site to mark my progress at a certain point. In some cases, it's mildly annoying, like an Ask that I think has one new answer but I've already seen it. In other cases, like fast-moving posts on The Blue (currently the "Hearings on 1/6" post), it makes it very hard to read some, go get some work done, and then come back to see what else has been said. Because it might drop me 30 or 40 posts back from where I've actually read to, and I have to skim to figure out where new stuff starts. If there's some workaround from my side, or something I'm doing wrong to cause this, I'd love to hear about it.

In the grand scheme of things, this is obviously not a major issue. But it has bugged me for long enough to finally at least ask about it....

posted by primethyme to Bugs at 11:05 PM (32 comments total)

For the first part of your question, this is almost always because there were comments deleted. So if the counter sees 7 new comments, but 3 were deleted before you clicked "show," you'd see only 3 comments. When the counter counts again, I believe it readjusts for the deleted comments, but before that happens it may says X where X is greater than the number of comments you see when clicking through.

About this, "it might drop me 30 or 40 posts back from where I've actually read to," I don't think I've experienced anything like this. Is it possible you might have had 2 or more browser windows open on the same thread? One is showing x new comments, but you've already refreshed in the other and read those comments? I'll ask frimble, but maybe this would account for that. Or maybe someone else has had this happen and can weigh in?
posted by taz (staff) at 11:25 PM on July 12, 2022 [1 favorite]

Actually, I'm only thinking of an open thread page, where you get the note at the bottom of the comments, but maybe you are talking about the main page. I understand this updating less well and yeah, it seems waaaay less accurate for me too. It's currently showing me all comments on every post as new, which is definitely not the case. I'm sure the deleted comments have an effect on this count as well, but I don't know how the counter sets or resets. I've asked frimble to look in here.
posted by taz (staff) at 11:47 PM on July 12, 2022 [1 favorite]

I have noticed that the Ukraine thread in the blue will say for example 240 comments and not list any as new. Then, when i open it, it becomes obvious that there have indeed been several new to me comments since i last opened it.
My feeling is this has not always been happening, only perhaps since one month. Not in the previous Ukraine thread.
I don't think it is deletions? Not many comments made in there currently and while i wish someone would delete the new Chomsky derail it has not happened.
posted by 15L06 at 1:44 AM on July 13, 2022 [1 favorite]

In other cases, like fast-moving posts on The Blue (currently the "Hearings on 1/6" post), it makes it very hard to read some, go get some work done, and then come back to see what else has been said.

In situations like this, I click the timestamp of the comment of the last comment I read. This leaves my browser tab 'view" at this spot (called a permalink). As long as I don't close that particular browser tab, I can just hit refresh and come back to that comment and start reading anew.
posted by Brandon Blatcher at 6:04 AM on July 13, 2022 [4 favorites]

You can also make the last comment you read a favorite, then look at the comments you favorited to click back quickly to where you were (as if, even, favorites were bookmarks!).
posted by chavenet at 6:14 AM on July 13, 2022 [1 favorite]

Sorry, maybe I didn't explain well. I think I assumed this was happening to everyone, but perhaps not.

Yes, leaving the tab open (ideally with the timestamp url of the last comment) does work. But if I close the tab and come back later, I have trouble. And yes I could bookmark/favorite/save it. But this happens all the time to me, and there are a lot of posts I come back to later to see if anything new has been added. I don't really want to bookmark them all or leave a ton of tabs open.

Here's the behavior I'm talking about:

1. I will click on a post (doesn't matter which metafilter site it is on) that has comments, and read all of the comments. Maybe while I am reading more will come in and I click the thing to load and read them, maybe not.
2. When I am finished reading all the current comments, I will close that tab.
3. Some time later, I go back to mefi, and I see that same post, saying something like "75 comments (20 new)."
4. I click on "20 new" to jump to the new comments.
5. I am taken to a post that I have already read. Sometimes ALL 20 of those "new" comments are ones I've already read, sometimes it's just a portion of them.
6. Regardless, I scroll down and find where the new stuff starts and read from there, then close the tab.
7. Repeat. Might come back and it says "78 comments (23 new)." Clearly not all 23 are new because last time I was here, it had 75 comments. Unless TONS were deleted and tons more came in to replace them.

This is sometimes more obvious on lower-volume posts, like on Ask. It might for hours say "3 comments (1 new)" when I know I have already read all three. I can click the "1 new" several times, come back later and it will still claim there is 1 new...

Where does it store the marker for how far I've read in a thread? Is it in a cookie? Stored server-side somewhere? Some other magic? FWIW I'm generally using Safari on Mac, if it's client-side and something weird is happening there.
posted by primethyme at 6:50 AM on July 13, 2022 [6 favorites]

LOL just now this post said "3 new," I clicked on it, and it took me to this url, which you can see goes to Brandon Blatcher's comment, two above mine. Indeed, there must be zero new (to me) because the last comment here is from me!
posted by primethyme at 6:55 AM on July 13, 2022 [1 favorite]

And I just went back to the metatalk main page and now it says "4 new" still linking to Brandon's coment. (Sorry, I know normally I shouldn't post three comments in a row, but it seemed relevant here...). I will note that shift-reload does not make a difference, so I don't think it's just cached (and the "new" number is incrementing each time anyway).
posted by primethyme at 6:57 AM on July 13, 2022

primethyme's experience is not unique and it's not new. I've found it happens quite commonly to me as well, and has for a long, long time.
posted by sardonyx at 6:57 AM on July 13, 2022 [18 favorites]

has for a long, long time

Indeed. Here's a metatalk from 2005.
posted by paper chromatographologist at 7:07 AM on July 13, 2022 [4 favorites]

That said, It's really no big deal to me. I'm not really bothered by it, but I didn't want to leave primethyme feeling alone in the world.
posted by sardonyx at 7:12 AM on July 13, 2022 [8 favorites]

Yeah I encounter this same thing as well, where the "(X new)" count from any of the main pages doesn't accurately reflect the actual number of new comments.

For stuff I really care about following closely, I will often use the workaround of clicking on the permalink for the last comment I read - and actually I do this every few comments as I'm working my way through a long thread on mobile because if I leave the tab open but switch to another app, the page might refresh on its own when I return, losing my place. Blerg.

It won't stop me from coming here, but it would be deluxe if the main page new counter worked more reliably so I wouldn't have to do the tab juggling thing!
posted by moonbeam at 8:00 AM on July 13, 2022 [1 favorite]

That 2005 post links to another about "20 minutes" but that link 404s. Can someone summarize what the 20 minutes thing is? It doesn't seem like it just updates the count/marker every 20 minutes because I'm pretty sure I've seen it be wrong for quite a bit longer than that, but I suppose I haven't timed it.
posted by primethyme at 8:31 AM on July 13, 2022

I'm not sure, but I think that the "x new" counter that appears on the main page is calculated from when you last loaded the main page, and doesn't actually reflect anything to do with threads you've clicked through to view or what comments you've actually seen. It's not quite as simple as that, because just refreshing the main page doesn't necessarily update the "x new" counters. For a while, I've suspected that it works based on some sort of timer that attempts to estimate a "browsing session," so if you return to the main page multiple times within some window of time, the "x new" counters don't update, but if you wait long enough (perhaps that's the 20 minutes referred to above?) between reloads then the site assumes it's a new visit and updates the counters. It's not a particularly effective estimate, but it seems like the sort of system that would have made sense 20+ years ago.
posted by biogeo at 9:10 AM on July 13, 2022 [5 favorites]

Oh yeah, I too think it counts based on visits to each main page (MF, Ask, Talk, etc), not to individual threads. If I'm keeping my eye on a few different threads on e.g., Ask, I'll sometimes avoid visiting the main Ask page so I don't disrupt the counters until I'm ready to check on updates from all the posts I'm interested in, lest I lose the "new" count. But even still, wonder if there is some timer like biogeo mentioned since the numbers still seem off a lot of the time?

Again it's not a huge deal to me, but I'm glad I haven't been imagining this miscount thing this whole time.
posted by moonbeam at 9:31 AM on July 13, 2022

A bit more explanation from pb in 2010.
posted by nobody at 10:26 AM on July 13, 2022 [6 favorites]

Drawing on the comment from pb posted above, it's worth noting that basically any solution that's more accurate than the one currently in use is going to involve Metafilter collecting a lot more information about exactly what every logged in user is reading and when they're reading it. This is more or less de rigeur for any modern professionally-built web site, but I know at least some people (me included) find the fact that Metafilter doesn't track that sort of thing as a significant part of its throw-back charm.
posted by firechicago at 10:55 AM on July 13, 2022 [15 favorites]

Maybe a simple "fix" is to adjust user expectations by having the text read "about x new" or "~x new". That way people know it's an estimate not to be relied on as an exact count. "About" is probably friendlier to the largest possible audience, "~" is more efficient use of space and slightly less distracting.
posted by biogeo at 11:33 AM on July 13, 2022 [1 favorite]

I think the tilde character doesn't read the same to people without that specific math background. The "X new" stuff will always be a bit off in fast-moving threads for the reasons pb mentioned way back when.
posted by jessamyn (staff) at 2:45 PM on July 13, 2022 [2 favorites]

Can we replace "x new comments" with "x lives changed"?
posted by rebent at 4:30 PM on July 13, 2022 [4 favorites]

Drawing on the comment from pb posted above, it's worth noting that basically any solution that's more accurate than the one currently in use is going to involve Metafilter collecting a lot more information about exactly what every logged in user is reading and when they're reading it.

FWIW, I don't think this has to be true. The most recent comment ID seen per thread could be saved locally, and the number of new comments calculated on the frontend. I don't think that's worth the extra complexity (and not working across devices/browsers), but it is possible.

That being said, doesn't Metafilter already have this information for as long as they retain HTTP logs? (which I assume they do, it's common practice basically everywhere I've worked, but I actually don't recall seeing documentation about what kinds of logging Metafilter does/how long it's retained for anywhere)

Anyway, I think fixing this would actually be pretty high-impact, but I understand there are like dozens of high-impact things to be fixed and this is probably pretty far down the list. It's something that I find annoying, though, and I would be sad if it stayed broken forever.
posted by wesleyac at 4:30 AM on July 14, 2022 [1 favorite]

Also, just from a technical perspective, I do want to mention that this reasoning from pb's post:

Storing which comments you've read vs. haven't doesn't scale as well to 10,000+ and we don't have giant server farms like Google.

was maybe true in 2010 when that was written, but definitely isn't generally true now — computers have gotten a lot bigger and faster in the past 12 years, and while I don't know what Metafilter's specific setup is, I did some benchmarking of 2010 commodity server hardware vs 2021 commodity server hardware recently and found that the DB workload I was looking at was more than 10x faster on the modern server, and that gap gets even larger if you look at high-end servers instead of commodity hardware. And in terms of storage, cost-per-gb has gone down like 75% (i guess like 60% inflation-adjusted).

It's pretty table-stakes these days for websites to have this feature (I can think of probably around half a dozen sites that do something like this off the top of my head), and I think it is important for Metafilter to modernize in ways like this with clear upside and very little downside.
posted by wesleyac at 5:10 AM on July 14, 2022

The most recent comment ID seen per thread could be saved locally, and the number of new comments calculated on the frontend. I don't think that's worth the extra complexity (and not working across devices/browsers), but it is possible.

This still implies the existence of a cookie containing the complete record of every thread you've ever read on the site. (As well as adding the complete list of all comment ids in every displayed post to the load of what's being compiled and sent every time the front page is loaded.)
posted by firechicago at 5:35 AM on July 14, 2022

This still implies the existence of a cookie containing the complete record of every thread you've ever read on the site.

Probably localStorage, rather than a cookie, so it wouldn't be sent to the server. IIRC localStorage is cleared at the same time browser history is, unless you specifically choose otherwise, so I wouldn't think there's a big privacy concern about this — people who are taking steps to make sure their browsing is not stored locally on their computer would not have to do anything differently to have that continue to be the case. I guess the worry would be that someone might delete a specific thread from their browser history and be surprised that it is still marked as read in the UI, but that feels like a fairly niche concern to me, and one that would be somewhat alleviated by making sure to expire the read/unread data when threads are closed (as well as when the user logs out, of course).

As well as adding the complete list of all comment ids in every displayed post to the load of what's being compiled and sent every time the front page is loaded

Yeah, this seems to me like a bigger concern. This could be limited by only having the 25 or so newest IDs, and just showing "25+ unread" in the case that there are more than that. I tried this on the current frontpage, and it looks like it adds 3,063 bytes gzipped, a 9% increase in page size. That's... more than I expected, but doesn't seem like a unreasonable tradeoff to have a unread count that actually shows what is unread, to me.

Details about how I got that number

This is from adding a data-comments="[comma separated list of newest 25 comment ids]" to each comments link in the post byline. I got the list of IDs via this bookmarklet:

javascript:(function()%7Blet%20ids%20%3D%20%5B%5D%3B%20for%20(const%20comment%20of%20document.getElementsByClassName(%22comments%22)%20)%20%7B%20try%20%7B%20ids.push(comment.querySelectorAll(%22.smallcopy%20a%22)%5B1%5D.href.split(%22%23%22)%5B1%5D)%20%7D%20catch%20(e)%20%7B%7D%20%7D%20%3B%20alert(ids.slice(-25).join())%7D)()

The frontpage was 131298 bytes before adding that and 139667 bytes after, which gziped to 33583 and 36646 respectively, using the default gzip options. This seems to be less well compressed than how Metafilter is actually served, since firefox tells me that the front page is 24kb transferred, so I'm not sure what's up with that — different compression options, maybe? 3kb should be a reasonable estimate of how much this would add, though.

posted by wesleyac at 7:26 AM on July 14, 2022 [1 favorite]

Yeah, speaking as a web developer, if I were interested in solving this more definitively, I'd make it a client-side script that tracks it in something like localStorage or indexedDB. The only reason to maybe try to involve a server would be if you want the "read-ness" of something to roam to other browsers/devices, but that feels like overkill for the amount of data you'd need to keep track of and the amount of processing overhead involved. Yes, computers and servers are more powerful, but that doesn't mean that you can just put a new feature like this on a high-traffic page and not consider the performance and cost ramifications. Plus, while MF generally feels like a good steward of my data, they're not immune to law enforcement requests, so the privacy implications of storing what you read on the servers would make that a non-starter in my opinion.
posted by Aleyn at 11:43 AM on July 14, 2022 [1 favorite]

I actually appreciate that it does track between devices, as long as I’m logged in. I can finish reading at night and then when I have downtime at work the next day I can pick up (approximately) where I left from a totally different machine and ip.

Which isn’t to say don’t explore other options but just to give my usage.
posted by one4themoment at 1:46 PM on July 14, 2022 [1 favorite]

I think removing them altogether should be on the table [...] They simply don’t work, and the more accurate XHR updates in the threads themselves make them partially redundant even if they did.

I actually find them very useful, flaky as they are. I often find it easier to just scroll down the main page than to keep individual thread tags open and then check them all one by one to see what's new (especially on mobile, where where the UI for handling tabs is awful in every browser I've tried), and even though the new comment counts are flaky as hell, they serve two purposes for me: they're one factor that reminds me to check back on some thread (I'm okay with that being in vain sometimes) and more importantly, they help reduce the need for scrolling because clicking on the new comments link takes me closer to the end of the thread if the new comment count is less than the total comment count. That doesn't make much difference in some threads but it's a big help in long threads, especially on mobile.
posted by trig at 10:04 PM on July 14, 2022 [6 favorites]

they help reduce the need for scrolling because clicking on the new comments link takes me closer to the end of the thread if the new comment count is less than the total comment count.
Yep. Handy.
posted by Don Pepino at 11:30 AM on July 15, 2022

You would only need to store the last read comment ID for each thread locally. Yeah, comments might be deleted, but comment IDs are monotonically increasing so the first unread is the first comment with ID greater than or equal to the stored ID.
posted by zixyer at 12:22 PM on July 15, 2022

Yup, this has been done (as mentioned in pb's comment). Plutor wrote a greasemonkey script that still works, I'm using it now with Tampermonkey in Vivaldi. I think this is the most recent version?

https://github.com/mefiscripts/mefiscripts/blob/master/userscripts/scroll_tag.user.js

It does actually track how far you've scrolled down a post and stores the last comment you viewed in a local cookie. Metafilter could ostensibly do the same thing in javascript without any additional stress on the servers.
posted by team lowkey at 10:07 PM on July 18, 2022

Nthing primethyme 's experience. Low key annoying, but still useful to get closer to the bottom of the thread than scrolling.

I would prefer not to see that feature removed, even if the performance doesn't change - when I say low key annoying I mean LOW!
posted by esoteric things at 8:39 PM on July 19, 2022

As a counterpoint, they definitely work for me everywhere and I find them charming and very useful. I am surprised to read that other people are having issues with them.
I’m adding this not to argue but just to present data showing that they are not broken for everyone.
posted by Vatnesine at 9:18 PM on July 19, 2022