Can anyone explain why metafilter won't display correct apostrophes and open/close quotes?

August 22, 2002 5:30 AM

Can anyone explain why metafilter won't display correct apostrophes and open/close quotes? I'm compulsive about them (I'm a graphic designer), they preview correctly, they work in normal html, but come up as question marks when posted.

some examples:

apostrophe: ’
open quote: “
close quote: ”

A suscpicion: maybe it's the spell check...

Thanks.

posted by jpburns to bugs at 5:30 AM (28 comments total)

There was another thread about this jpburns, but I'll be fecked if I can find it.
posted by Summer at 5:35 AM on August 22, 2002

Found it!
posted by Summer at 5:37 AM on August 22, 2002

Ah. That caught me up on the term "tomcatspikish," but still didn't address the problem of metafilter not passing proper punctuation. There are some posts from Matt about how he was aware of the problem, and had some scheme to fix it, but, here we are with the same problem.
It's not a big problem, and not all that important in the grand scheme of things, but personally annoying...

Thanks anyway.

posted by jpburns at 6:10 AM on August 22, 2002

Matt said something about high-ASCII characters that the Java engine of ColdFusion MX does not like. Specifically, if you compose your notes in anything but notepad or the comments box, the "smart quotes" will look fine on preview and die once you post.

posted by Irontom at 6:43 AM on August 22, 2002

but if you are a graphic designer, you should know that there are no such chars in html. that's a microsoft hack.
posted by quonsar at 6:53 AM on August 22, 2002

A suscpicion: maybe it's the spell check...

Don't tell me that's fucked up as well!
posted by MiguelCardoso at 7:22 AM on August 22, 2002

Entity Test

“

”

‘

’
posted by alan at 7:59 AM on August 22, 2002

Entity Test #2

“

”

‘

’
posted by alan at 8:04 AM on August 22, 2002

here's something I've never noticed before... in this thread (which unfortunately is a double post and will soon be deleted), there are smart quotes in the post that work on the main page, but are boxes when the post is viewed at the top of the comments page. what's up with that?
posted by GeekAnimator at 8:13 AM on August 22, 2002

I've gotten them after composing something in the edit box.

Also, 'tomcatspikisms' would seem to be a thing of the past, as apparently, and disappointingly, his 'difficulties' with english were merely a sub-JeffK (or, alternately, the corresponding Incoherent Guy over at Plastic.com who I can't be bothered trying to look up) attention-getting device.
posted by stavrosthewonderchicken at 8:18 AM on August 22, 2002

but if you are a graphic designer, you should know that there are no such chars in html. that's a microsoft hack.

sure there are, quonsar. the ldquo and rdquo character entities correspond to the left- and right-double quote symbols, while lsquo and rsquo correspond to the single quotes. maybe they weren't always a part of the specification, but as of html 4, they are.

how will coldfusion handle unicode if it can't handle the eighth bit of ascii?
posted by moz at 8:20 AM on August 22, 2002

thanks moz. i figured i musta missed something.
posted by quonsar at 8:34 AM on August 22, 2002

If proper quotation is that important to you, you can manually use HTML entities.

Type your post using the proper entity instead of quotation marks. Before you preview, copy your post to the clipboard. Preview. Once everything looks ok, delete your entire post and paste the copy from your clipboard (may not be necessary on metafilter proper, but it’s best to be safe). Then post and you’ll have magically typographically correct quotes.

posted by alan at 8:37 AM on August 22, 2002

jpburns, here's how I do character entities:

1. Type in your character entity codes (i.e., ampersand codes) in your post.
2. Preview your post. You should see the characters you want in preview but if you look in the text box. you'll see that your character entity codes have been replaced.
3. After you've completed previewing, spell-checking and editing, go back into the text box and re-enter the character entity codes.
4. Now post.

This may not work when posting a front-page link, but it works for comments.

–‘’“”—˜™©®
posted by timeistight at 8:37 AM on August 22, 2002

I'm still working on this, this is an continued problem with ColdFusion's database connector. It doesn't like high-ascii junk.
posted by mathowie at 10:04 AM on August 22, 2002

You shouldn't have problems if you always convert any special characters/symbols into entities before storing the HTML. If you don't want to allow Unicode (probably no reason to exclude it unless you're rigorously sticking to ISO) you can convert common characters into ISO-8859-1 approximations (e.g. curly quotes to straight quotes), but at its simplest the conversion just involves changing any characters with high values into numeric entities (leave anything less than 127 alone, convert greater than 160 to numeric entities, junk anything in-between or convert to appropriate alternatives; not absolutely certain that's right, it's off the top of my head).

posted by malevolent at 10:40 AM on August 22, 2002

Is it possible to preprocess the post before you send it to the database? So if it finds the high ascii stuff, it will convert it to the reasonable equivalent. It doesn't fix the core problems, but it seems more user friendly than trying to tell everyone to turn off smart quotes in Word.

(on preview: what malevolent said. sort of.)
posted by Gary at 10:44 AM on August 22, 2002

more user friendly than trying to tell everyone to turn off smart quotes in Word.
i STILL don't beleive people actually load word to compose a mefi post or an email. do they also use high-explosive anti-tank projectiles as bug repellant? hammers to seat screws? air hammers to knock on doors? boggles my mind, what there is of it.
posted by quonsar at 11:11 AM on August 22, 2002

I always have WordPerfect running, usually with two documents open - what I'm working on, and what I'm "working" on, if you get my drift.
posted by yhbc at 11:22 AM on August 22, 2002

You shouldn't have problems if you always convert any special characters/symbols into entities before storing the HTML.

Except there are still problems with ColdFusion MX; even just doing a search-and-replace on the form submission before it ever touches the database, the high-ASCII chars get all munged up.

It doesn't make any sense to me either, but as it is a gnarly (and pressing) problem for me at my day job, I feel Matt's pain.
posted by Sapphireblue at 11:24 AM on August 22, 2002

i STILL don't beleive people actually load word to compose a mefi post or an email. do they also use high-explosive anti-tank projectiles as bug repellant? hammers to seat screws? air hammers to knock on doors? boggles my mind, what there is of it.

The spell checker in Word is still better than any web-based spell checker I’ve seen. And good spelling helps people focus on your post instead how you decided to spell "believe". More importantly, it helps keep the spelling-nazis from detouring threads. Just like fixing the box problem will prevent threads from being hijacked and new metatalk threads from being created every other week about where the stupid boxes came from that weren't there on preview.
posted by Gary at 11:28 AM on August 22, 2002

D'oh
posted by Gary at 11:30 AM on August 22, 2002

Is it really that difficult to replace whichever funky characters you're in love with with their numeric equivalents? Here's some to get you started:
apostrophe: (’) ’
open quote: (“) “
close quote: (”) ”
open single quote: (‘) ‘
em-dash: (—) —
en-dash: (–) –
posted by timeistight at 11:32 AM on August 22, 2002

I always have WordPerfect running, usually with two documents open - what I'm working on, and what I'm "working" on, if you get my drift.
i bleeve i do!
posted by quonsar at 12:48 PM on August 22, 2002

Is it really that difficult to replace whichever funky characters you're in love with with their numeric equivalents?

Apparently it is that difficult.
posted by timeistight at 1:24 PM on August 22, 2002

The spell checker in Word is still better than any web-based spell checker I've seen.

Haven't seen the one in OmniWeb, I reckon. (It uses the spelling service built into Mac OS X, and squiggly-underlines any potentially misspelled words it finds in any textarea.) I have no idea why Microsoft doesn't do this in IE; they have the code, it's got to be just a matter of plugging it in.
posted by kindall at 2:51 PM on August 22, 2002

It's not just about Word's spell checker. You can get this problem if you copy text in the browser that contains a smart quote and then paste it into MeFi's comment box. I posted a comment that looked OK on preview and had munged quotes once it was posted (sure points to the database as the culprit).

posted by fuzz at 5:50 PM on August 22, 2002

Except there are still problems with ColdFusion MX; even just doing a search-and-replace on the form submission before it ever touches the database, the high-ASCII chars get all munged up.

Ah right, sounds like it's not at all Unicode-friendly. Another option is to do the conversion with JavaScript; I've got a simple utility for marking up text that handles ISO/Unicode characters pretty well, if anyone wants to nick bits of the scripting feel free (although I'd appreciate being told what you're using it for, just for my own curiosity, and if you use most of the code unmodified a credit would be polite).

posted by malevolent at 1:46 AM on August 23, 2002