The tag goes inside the tag, which can take multiple tags, but no tags. Understand?

August 17, 2009 11:13 PM

Pony request: proper escaped HTML entity handling in memail.

I just sent a totally nonsensical memail to somebody talking about an HTML technique.

I used a whole bunch of HTML entities for my <>'s (written &lt; and &gt; respectively). And then I previewed my message. It looked great up in the preview area.

What I didn't notice was that the actual text box (which is what's sent) contained the literal equivalents of my escaped entities. Which meant that, thanks to memail's HTML stripper, I sent a memail with giant goddamn holes all throughout it.

Couldn't memail just, you know, preserve HTML entities and leave it up to the browser to render them? None of the other text entry boxes on the site do such unkosher things to my text.

posted by Netzapper to Feature Requests at 11:13 PM (40 comments total)

NO.
posted by crossoverman at 11:20 PM on August 17, 2009

Think of the children.
posted by peeedro at 11:32 PM on August 17, 2009

When you started the memail, did you fail to see this:
"Note: HTML will be stripped from your message. If you want to include a link, put a URL on a line by itself."?
posted by Cranberry at 11:42 PM on August 17, 2009

I think you're misinterpreting my request, Cranberry.

I don't want to put HTML in the message. I want to be able to talk about HTML in the message by using HTML entities.

So, I wrote: "The &lt;blink&gt; tag makes things blink".

With the intention that the browser would, on viewing, turn that into: "The <blink> tag makes things blink". [Note that you can see the brackets and the tag; your browser is not interpreting it as HTML.] And, if you don't preview, that works just perfectly.

However, if you preview, the text box at the bottom says: "The <blink> tag makes things blink", which is not what I typed, but contains the literal values of the HTML entities (as the browser would render them).

Then, when I send it, the newly-formed, now-unescaped HTML is stripped. And my message comes through as: "The tag makes things blink."

I don't want the HTML stripper turned off... I want the resolution of HTML entities turned off in preview. If I type "&lt;", that's what should show up in the text box after preview, not "<".
posted by Netzapper at 11:57 PM on August 17, 2009

[Incidentally, it's kind of annoying that you can't send messages to yourself to test formatting. But , I guess I'll live with that.]
posted by Netzapper at 12:02 AM on August 18, 2009

Netzapper, I gave you my e-mail address... you're welcome to e-mail me any time instead of Mefi Mail. :) Thanks for the help.
posted by IndigoRain at 12:13 AM on August 18, 2009

I'd love to be able to send all kinds of crazy shit through memail, but I accept that I can't.

I don't accept that the preview doesn't show me what the message is going to look like. That's bollocks.
posted by aubilenon at 1:52 AM on August 18, 2009

No, that's the thing! You can send the HTML entities. This isn't the scrubbing doing its thing. Furthermore, the preview accurately reflects what you type the first time.

I repeat! I am not talking about the HTML scrubbing! This is a bug report, not a request to change intentional behavior! (At least I assume it isn't intentional behavior.)

It's just that when you hit preview, HTML entity references get transformed in the text box. This isn't inherently a problem most of the time. No part of the system cares if "&frac12;" is that proper glyph or spelled out as "&frac12;".

BUT! Let's say you've used HTML entities to escape the angle brackets around HTML code so that they will be displayed instead of being treated as code (<blink>like this</blink>), so that you can discuss the <blink> tag with somebody, just like I'm doing right here. If the preview is correct, and you then hit send without going back into the text box and replacing all the angle brackets with their entity references, then all of your carefully-escaped-so-they-will-neither-offend-nor-be-scrubbed tag text is eaten by the scrubber. This totally defeats the purpose of preview, since you'd have to go back and re-edit your message to make it correct anyway.

Let me try to make this simpler:

I start a new message and I type the following into the message box: "&lt;blink&gt;". Then I hit preview. The preview div at the top of the page will properly display "<blink>". However, the box at the bottom of the screen will be populated with "<blink>", when it should still display what I typed in the first place, which, I remind you, is "&lt;blink&gt;" This is the faulty behavior.

If I then press send, the text gets scrubbed, because the angle brackets were no longer escaped, but literal characters. If I never hit preview, things are good... but, then I can't preview my message before sending it. This is the reason that the faulty behavior has unfortunate consequences, but is not faulty behavior itself.

Even putting aside all of the big, scary, and apparently confusing words like "entity" and "reference", it's like if I typed "colour" into the text box, pressed preview, and it was transformed to "color" even before I send it. There's no reason for it to be changed... and yet it is.
posted by Netzapper at 2:32 AM on August 18, 2009

[Huh... that's real weird. The one-half fraction entity isn't rendered here. And yet it's rendered in live preview... and it's rendered by my browser normally. What's the logic behind that? Why can I have angle brackets but no fraction?]
posted by Netzapper at 2:35 AM on August 18, 2009

Metafilter having completely screwy handling of escaping HTML and lying to you about it in preview (and not even returning the same text in the input box on the preview page) is something of a tradition here. Asking for it to be fixed is like asking for a professional white background.
posted by cillit bang at 2:41 AM on August 18, 2009

This reminds me of when I was in school and the other kids were talking about some other kid's birthday party that I wasn't invited to.
posted by chillmost at 2:42 AM on August 18, 2009 [1 favorite]

Man, that was a great meetup we had in chillmost's town.
posted by TheophileEscargot at 3:06 AM on August 18, 2009 [6 favorites]

Since the mods aren't awake yet, I'll give you some probable responses:

1. MeMail is not intended as a replacement for GMail, etc.
2. MeMail is not a priority
3. For heavy-duty, fully-featured email, use real email
4. pb is very busy
5. pb thinks that MeMail was a pretty good April Fools joke, but it is time to pull the plug

posted by double block and bleed at 3:07 AM on August 18, 2009 [2 favorites]

The aggressive featurelessness of memail is a feature, not a bug.
posted by DU at 4:16 AM on August 18, 2009

Also, the preview-fucking-up-formatting thing is a long-standing bug feature.
posted by dg at 4:25 AM on August 18, 2009

I've got an idea for a toy based on My little ponies, but targeted at active, sports conscious, prep fashion aware boys. I'm calling it the My Little Bronie.
posted by boo_radley at 5:30 AM on August 18, 2009

This is one of those bugs that would be great to use the Contact form for.
posted by smackfu at 5:30 AM on August 18, 2009

Can we make the colorshifting send me a memail announcing new products when it gets to blue? It could be like an acid pony. Or a trippy pony. Or a trippy hippy! treepy mippymail. the colors!
*whirr*
posted by Cat Pie Hurts at 5:59 AM on August 18, 2009

I'm pretty sure this will not happen. I understand you, Netzapper, but if you want to talk about HTML entities you'll have to use Real Mail for that. Or a phone call.
posted by jessamyn (staff) at 7:00 AM on August 18, 2009

...if you want to talk about HTML entities you'll have to use Real Mail for that. Or a phone call.

...or skip the preview and trust that you have it right the first time. Or preview in your browser by composing the memail in notepad, saving it as an HTML file, and opening it in Firefox or Internet Explorer or whatever.

It sucks that preview screws up HTML entities in all text boxes across the site, but it especially sucks that it screws them up in memail.
posted by muddgirl at 7:31 AM on August 18, 2009

If I'm hearing right, it sounds like there's two specific complaints:

1. The mefimail Preview button is replacing non-stripped html entities with their literal (and hence strippable) glyphs, causing those to get scrapped out of mefimails that have been previewed even though they'd send fine without a preview.

2. Not all named entities (e.g. ½) show up on mefi despite showing up in other web venues and being displayed in Live Preview in thread.

Problem number one may indeed be a bug, as stated; I think where we are on that stuff is that we used to have the preview-munges-entities problem sitewide for a long time but then pb and Matt managed to fix it. It's possible then that mefimail preview just didn't get that same bit of love, in which case this improving is possible. But let me know if I'm misunderstanding the nature of the problem.

Problem number two is, I believe, a combination of our whitelist-only approach to named entities and the fact that Live Preview is funky old third-party code that's out of step with what the site actual permits and is a pain in the ass to fix. When in doubt on complex html shenanigans in actual comment threads do a real (and non-destructive, yay!) Preview and check your shit that way.
posted by cortex (staff) at 7:53 AM on August 18, 2009

Just set MeFi Mail so the form will preserve HTML entities on preview. But keep in mind that MeFi Mail often also goes out as email, and we don't send that out as HTML-email. So when people first see a message with HTML entities in their email client, they'll see the bare entities, not the character you intended to show. It is a pain to explain HTML in a medium that uses HTML for display. That will always be a headache. So it'd be good to jump over to a tool that is made specifically for sharing snippets of HTML and link to your code instead of sending it along. Or link to a generic page about the blink tag, stuff in our FAQ, or something like that.
posted by pb (staff) at 7:59 AM on August 18, 2009

It turns out that I was taking the &frac12; example on faith for no good reason, since it does indeed seem to display. And in fact it may be I just misremembered on that front, and that named entities in general are fine and we're actually just filtering out numeric character references (e.g. 189; gets rendered as 189; and xBD; as xBD, while &frac12; becomes &frac12;).
posted by cortex (staff) at 8:00 AM on August 18, 2009

Though, heh, that went badly too. Hmm.
posted by cortex (staff) at 8:00 AM on August 18, 2009

I've learned my lesson to stay far, far away from HTML entities. I've gone so far as to swear off numbers.

and now im giving up punctuation and capitals just in case metafilter decides to make me look like a doofus by stripping out all periods and capital letters
posted by Plutor at 8:09 AM on August 18, 2009

&frac12; and other named fraction entities should now be working in live preview and HTML preview.
posted by pb (staff) at 8:13 AM on August 18, 2009

&frac12; ½

[Preview]

Kick ass, pb.
posted by cortex (staff) at 8:22 AM on August 18, 2009

CENSORSHIP!!

Seriously though, Metafilter has never played nicely with html (encoded, escaped or litteral). It would be nice to implement a real text-edit field for posts, comments, and mail.
posted by blue_beetle at 8:31 AM on August 18, 2009

So when people first see a message with HTML entities in their email client, they'll see the bare entities, not the character you intended to show.

Could you not go the extra step and decode the entities? The expectation surely is that any given message displays the same way in all venues (and matches preview), and it seems like you're only one call to entity_decode (or equivalent) from achieving that.
posted by cillit bang at 10:54 AM on August 18, 2009

The browser does the work of decoding entities here at the site. But we could theoretically write a function to transform any HTML entity into its character equivalent for the email portion of MeFi Mail. But once you add in that new function, you introduce potential other parsing problems. We say it so much that it's a joke, but we're trying to keep MeFi Mail as simple as possible. We're not in the email business, and I'm worried that trying to get fancy with entities will cause more problems than the edge cases it solves.
posted by pb (staff) at 10:59 AM on August 18, 2009

It's amusing that not supporting entities will require the parser to convert some characters to entities.
posted by smackfu at 11:08 AM on August 18, 2009

Q: Why did the chicken cross the road?
A: We're trying to keep MeFi Mail as simple as possible.
posted by Plutor at 11:12 AM on August 18, 2009 [4 favorites]

The browser does the work of decoding entities here at the site.

Well you could go the opposite way and escape messages when they're sent to the browser, which would also be more secure.
posted by cillit bang at 11:38 AM on August 18, 2009 [1 favorite]

YAY! WOOOO!

Thank you so much pb! You have returned sanity to my life! and restored equilibrium to the universe!

it's clear that 'pb' stands for "particularly badass".
posted by Netzapper at 12:31 PM on August 18, 2009

it's clear that 'pb' stands for "particularly badass".

*phew* I can finally stop feeling sad because he has no jelly.
posted by Cat Pie Hurts at 1:30 PM on August 18, 2009 [1 favorite]

I am amused by how many people think a legitimate (if minor) bug should stay as-is because they're used to it themselves (ie "that's a long standing bugfeature" and similar.) We see the same resistance in usability testing, where the painfully awkward pre-existing design pattern has immense inertia over the much-improved new one, even in cases where the pre-existing design pattern has only been seen in development by developers -- sometimes people who have grown used to it in development will claim that users can "get used to it too" when we launch rather than being open to re-learning it. Human nature, I guess.
posted by davejay at 2:52 PM on August 18, 2009 [2 favorites]

I'm gonna start using "has no jelly" as a colorful way of saying "lacks bad-ass-itude."
posted by nebulawindphone at 8:30 PM on August 18, 2009 [1 favorite]

(Or would that be "ass badness"?)
posted by nebulawindphone at 8:30 PM on August 18, 2009

ASS QUALITY UNUSUALLY HIGH STOP
PLEASE SEND JELLY STOP
posted by nebulawindphone at 8:31 PM on August 18, 2009 [4 favorites]

I am amused by how many people think a legitimate (if minor) bug should stay as-is because they're used to it themselves (ie "that's a long standing bugfeature" and similar.)...
I didn't say it was legitimate, I just said it was long-standing.
posted by dg at 3:45 AM on August 19, 2009