Comments on: Can anyone explain why metafilter won't display correct apostrophes and open/close quotes?

Can anyone explain why metafilter won't display correct apostrophes and open/close quotes?

jpburns — Thu, 22 Aug 2002 05:30:24 -0800

Can anyone explain why metafilter won't display correct apostrophes and open/close quotes? I'm compulsive about them (I'm a graphic designer), they preview correctly, they work in normal html, but come up as question marks when posted.

some examples:

apostrophe: '
open quote: "
close quote: "

A suscpicion: maybe it's the spell check...

Thanks.

By: Summer

Summer — Thu, 22 Aug 2002 05:35:51 -0800

There was another thread about this jpburns, but I'll be fecked if I can find it.

By: Summer

Summer — Thu, 22 Aug 2002 05:37:42 -0800

Found it!

By: jpburns

jpburns — Thu, 22 Aug 2002 06:10:31 -0800

Ah. That caught me up on the term "tomcatspikish," but still didn't address the problem of metafilter not passing proper punctuation. There are some posts from Matt about how he was aware of the problem, and had some scheme to fix it, but, here we are with the same problem.
It's not a big problem, and not all that important in the grand scheme of things, but personally annoying...

Thanks anyway.

By: Irontom

Irontom — Thu, 22 Aug 2002 06:43:01 -0800

Matt said something about high-ASCII characters that the Java engine of ColdFusion MX does not like. Specifically, if you compose your notes in anything but notepad or the comments box, the "smart quotes" will look fine on preview and die once you post.

By: quonsar

quonsar — Thu, 22 Aug 2002 06:53:35 -0800

but if you are a graphic designer, you should know that there are no such chars in html. that's a microsoft hack.

By: MiguelCardoso

MiguelCardoso — Thu, 22 Aug 2002 07:22:42 -0800

A suscpicion: maybe it's the spell check...

Don't tell me that's fucked up as well!

By: alan

alan — Thu, 22 Aug 2002 07:59:12 -0800

Entity Test

“

”

‘

’

By: alan

alan — Thu, 22 Aug 2002 08:04:08 -0800

Entity Test #2

"

"

'

'

By: GeekAnimator

GeekAnimator — Thu, 22 Aug 2002 08:13:41 -0800

here's something I've never noticed before... in this thread (which unfortunately is a double post and will soon be deleted), there are smart quotes in the post that work on the main page, but are boxes when the post is viewed at the top of the comments page. what's up with that?

By: stavrosthewonderchicken

stavrosthewonderchicken — Thu, 22 Aug 2002 08:18:33 -0800

I've gotten them after composing something in the edit box.

Also, 'tomcatspikisms' would seem to be a thing of the past, as apparently, and disappointingly, his 'difficulties' with english were merely a sub-JeffK (or, alternately, the corresponding Incoherent Guy over at Plastic.com who I can't be bothered trying to look up) attention-getting device.

By: moz

moz — Thu, 22 Aug 2002 08:20:09 -0800

but if you are a graphic designer, you should know that there are no such chars in html. that's a microsoft hack.

sure there are, quonsar. the ldquo and rdquo character entities correspond to the left- and right-double quote symbols, while lsquo and rsquo correspond to the single quotes. maybe they weren't always a part of the specification, but as of html 4, they are.

how will coldfusion handle unicode if it can't handle the eighth bit of ascii?

By: quonsar

quonsar — Thu, 22 Aug 2002 08:34:18 -0800

thanks moz. i figured i musta missed something.

By: alan

alan — Thu, 22 Aug 2002 08:37:22 -0800

If proper quotation is that important to you, you can manually use HTML entities.

Type your post using the proper entity instead of quotation marks. Before you preview, copy your post to the clipboard. Preview. Once everything looks ok, delete your entire post and paste the copy from your clipboard (may not be necessary on metafilter proper, but it’s best to be safe). Then post and you’ll have magically typographically correct quotes.

By: timeistight

timeistight — Thu, 22 Aug 2002 08:37:40 -0800

jpburns, here's how I do character entities:

Type in your character entity codes (i.e., ampersand codes) in your post.
Preview your post. You should see the characters you want in preview but if you look in the text box. you'll see that your character entity codes have been replaced.
After you've completed previewing, spell-checking and editing, go back into the text box and re-enter the character entity codes.
Now post.

This may not work when posting a front-page link, but it works for comments.

–‘’“”—˜™©®

By: mathowie

mathowie — Thu, 22 Aug 2002 10:04:05 -0800

I'm still working on this, this is an continued problem with ColdFusion's database connector. It doesn't like high-ascii junk.

By: malevolent

malevolent — Thu, 22 Aug 2002 10:40:21 -0800

You shouldn't have problems if you always convert any special characters/symbols into entities before storing the HTML. If you don't want to allow Unicode (probably no reason to exclude it unless you're rigorously sticking to ISO) you can convert common characters into ISO-8859-1 approximations (e.g. curly quotes to straight quotes), but at its simplest the conversion just involves changing any characters with high values into numeric entities (leave anything less than 127 alone, convert greater than 160 to numeric entities, junk anything in-between or convert to appropriate alternatives; not absolutely certain that's right, it's off the top of my head).

By: Gary

Gary — Thu, 22 Aug 2002 10:44:08 -0800

Is it possible to preprocess the post before you send it to the database? So if it finds the high ascii stuff, it will convert it to the reasonable equivalent. It doesn't fix the core problems, but it seems more user friendly than trying to tell everyone to turn off smart quotes in Word.

(on preview: what malevolent said. sort of.)

By: quonsar

quonsar — Thu, 22 Aug 2002 11:11:47 -0800

more user friendly than trying to tell everyone to turn off smart quotes in Word.
i STILL don't beleive people actually load word to compose a mefi post or an email. do they also use high-explosive anti-tank projectiles as bug repellant? hammers to seat screws? air hammers to knock on doors? boggles my mind, what there is of it.

By: yhbc

yhbc — Thu, 22 Aug 2002 11:22:54 -0800

I always have WordPerfect running, usually with two documents open - what I'm working on, and what I'm "working" on, if you get my drift.

By: Sapphireblue

Sapphireblue — Thu, 22 Aug 2002 11:24:07 -0800

You shouldn't have problems if you always convert any special characters/symbols into entities before storing the HTML.

Except there are still problems with ColdFusion MX; even just doing a search-and-replace on the form submission before it ever touches the database, the high-ASCII chars get all munged up.

It doesn't make any sense to me either, but as it is a gnarly (and pressing) problem for me at my day job, I feel Matt's pain.

By: Gary

Gary — Thu, 22 Aug 2002 11:28:36 -0800

i STILL don't beleive people actually load word to compose a mefi post or an email. do they also use high-explosive anti-tank projectiles as bug repellant? hammers to seat screws? air hammers to knock on doors? boggles my mind, what there is of it.

The spell checker in Word is still better than any web-based spell checker I've seen. And good spelling helps people focus on your post instead how you decided to spell "believe". More importantly, it helps keep the spelling-nazis from detouring threads. Just like fixing the box problem will prevent threads from being hijacked and new metatalk threads from being created every other week about where the stupid boxes came from that weren't there on preview.

By: Gary

Gary — Thu, 22 Aug 2002 11:30:35 -0800

D'oh

By: timeistight

timeistight — Thu, 22 Aug 2002 11:32:20 -0800

Is it really that difficult to replace whichever funky characters you're in love with with their numeric equivalents? Here's some to get you started:
apostrophe: (’) 
open quote: (“) 
close quote: (”) 
open single quote: (‘) 
em-dash: (—) 
en-dash: (–) 

By: quonsar

quonsar — Thu, 22 Aug 2002 12:48:21 -0800

I always have WordPerfect running, usually with two documents open - what I'm working on, and what I'm "working" on, if you get my drift.
i bleeve i do!

By: timeistight

timeistight — Thu, 22 Aug 2002 13:24:56 -0800

Is it really that difficult to replace whichever funky characters you're in love with with their numeric equivalents?

Apparently it is that difficult.

By: kindall

kindall — Thu, 22 Aug 2002 14:51:42 -0800

The spell checker in Word is still better than any web-based spell checker I've seen.

Haven't seen the one in OmniWeb, I reckon. (It uses the spelling service built into Mac OS X, and squiggly-underlines any potentially misspelled words it finds in any textarea.) I have no idea why Microsoft doesn't do this in IE; they have the code, it's got to be just a matter of plugging it in.

By: fuzz

fuzz — Thu, 22 Aug 2002 17:50:19 -0800

It's not just about Word's spell checker. You can get this problem if you copy text in the browser that contains a smart quote and then paste it into MeFi's comment box. I posted a comment that looked OK on preview and had munged quotes once it was posted (sure points to the database as the culprit).

By: malevolent

malevolent — Fri, 23 Aug 2002 01:46:31 -0800

Except there are still problems with ColdFusion MX; even just doing a search-and-replace on the form submission before it ever touches the database, the high-ASCII chars get all munged up.

Ah right, sounds like it's not at all Unicode-friendly. Another option is to do the conversion with JavaScript; I've got a simple utility for marking up text that handles ISO/Unicode characters pretty well, if anyone wants to nick bits of the scripting feel free (although I'd appreciate being told what you're using it for, just for my own curiosity, and if you use most of the code unmodified a credit would be polite).