Comments on: Pi fine on preview bad on post

Pi fine on preview bad on post

jfuller — Tue, 25 Nov 2003 15:32:00 -0800

The other day I posted a sentence including the Greek letter pi. This appeared as the Greek letter on preview but came out as a couple of question marks in the actual post. Today a post by languagehat included some Kanji characters and a bit later in the thread he says, darn it, ...those characters looked fine on preview. Can someone explain the difference between preview and live posting when it comes to special characters? Thanks!

By: elwoodwiles

elwoodwiles — Tue, 25 Nov 2003 15:52:27 -0800

Someone will come along and explain this better, but here I go anyway. Evidently there is something in the coding of MeFi's comment box that kicks out certain HTML for certain symbols. If you code HTML into a comment, say for example:

I ♥ pancakes!! it will show the heart on preview. If you look in the comment box while previewing you'll even see the characters in the preview box. But the preview box is full of lies. The solution is to write the HTML initially, check it on preview, delete the HTML character and then write the HTML again on preview before you post.

By: elwoodwiles

elwoodwiles — Tue, 25 Nov 2003 15:57:36 -0800

I ? pancakes

By: gleuschk

gleuschk — Tue, 25 Nov 2003 15:57:56 -0800

MetaFilter: the preview box is full of lies.

By: holloway

holloway — Tue, 25 Nov 2003 16:04:02 -0800

Special characters are unicode characters, and there's a difference between UTF-8 encoded text (the bytes) and character entities (ASCII strings like &).

If it was inserted by copy/pasting the character in then it's probably UTF-8. If the person typed the character entity then it's (obviously) a character entity.

UTF-8 could be broken if there was something in the code that wants ASCII or 8879 or something, and breaks on UTF-8 (a while ago Matt mentioned something about a UTF-8 problem with JDBC/ColdFusion MX). The is different code involved in a post preview and a post submission. Preview probably just feeds back your form submission into the webpage, and doesn't touch the database. If there was a problem with UTF-8 encoded text and the database you wouldn't see the break until it went through the database (obviously) and it was live.

Character entities are broken in metafilter because it doesn't reencode ampersands into & for the form box, which is a simple thing to fix.

By: dg

dg — Tue, 25 Nov 2003 16:29:41 -0800

Also discussed here, here, here, here, here, here, here, here, here and here.

By: holloway

holloway — Tue, 25 Nov 2003 16:33:56 -0800

Crap, there were a few of those I didn't post in.

By: mathowie

mathowie — Tue, 25 Nov 2003 16:40:23 -0800

I think it's mostly an issue with the JDBC/CFMX connection, since I'm pretty sure I'm storing stuff in MSSQL as utf-8 and I specify utf-8 for all form submissions (which is why it works in preview). It's when the posts head to the database that there is trouble.

By: holloway

holloway — Tue, 25 Nov 2003 16:55:08 -0800

To fix the entity bug the bit of code that writes between <textarea> needs to replace & with & to keep the entities rather than resolving them, and then that way previews won't resolve entities into utf-8 data.

Won't fix the utf-8 db problem, but then entities will be consistant.

(heh, you just changed it?)

By: scarabic

scarabic — Tue, 25 Nov 2003 16:56:37 -0800

I want to ? you in the ?

By: scarabic

scarabic — Tue, 25 Nov 2003 16:56:45 -0800

damn! get right on this!

By: holloway

holloway — Tue, 25 Nov 2003 16:57:51 -0800

Oh, so there's a replacement for <lt; to make it &lt; and >gt; to make it &gt; , but not & to & ?

By: holloway

holloway — Tue, 25 Nov 2003 16:58:17 -0800

*cries*

By: mischief

mischief — Tue, 25 Nov 2003 17:06:47 -0800

The obvious solution is to use English exclusively and quit trying to be cute/pretentious/know-it-all. ;-P

By: MiguelCardoso

MiguelCardoso — Tue, 25 Nov 2003 18:04:45 -0800

This classy, classic tutorial from much-missed Evanizer is very useful.

By: zpousman

zpousman — Tue, 25 Nov 2003 20:14:29 -0800

I *heart* evanizer.

Fuck that up mr. metafilter textbox man!

By: gen

gen — Tue, 25 Nov 2003 23:08:30 -0800

€

By: gen

gen — Tue, 25 Nov 2003 23:08:52 -0800

well, lookit...it works.

By: i_cola

i_cola — Wed, 26 Nov 2003 02:05:25 -0800

'I think it's mostly an issue with the JDBC/CFMX connection, since I'm pretty sure I'm storing stuff in MSSQL as utf-8 and I specify utf-8 for all form submissions'

WHAT.
THE.
FUCK.
MATT?

;-)

By: chill

chill — Wed, 26 Nov 2003 03:04:11 -0800

I'm pretty sure I'm storing stuff in MSSQL as utf-8 and I specify utf-8 for all form submissions
The character set utf-8 isn't available in MSSQL, if you want to work in UTF-8 you have to store the data in fields with a Unicode data type (nvarchar etc) and convert to and from UTF-8 on the way in and out. The way I work in an ASP/SQL Server environment is pretty simple...
Form pages come in as UTF-8 (assuming someone hasn't changed the character set for some reason), and the ASP locale is set to UTF-8. The text fields in the database are nvarchar. Because the server knows that the string is UTF-8 and the field it is going to put it in is Unicode, the conversion is carried out automatically. conversely, when you retrieve data from the database, it knows that the data is unicode and that it has to convert it to UTF-8 when you do a response.write.

By: arto

arto — Wed, 26 Nov 2003 04:34:18 -0800

No, i_cola, that's WTF-M@ encoding.

HTH.

By: SweetJesus

SweetJesus — Wed, 26 Nov 2003 10:36:08 -0800

?????!

By: jfuller

jfuller — Wed, 26 Nov 2003 11:03:14 -0800

> I'm pretty sure I'm storing stuff in MSSQL as utf-8 and I specify utf-8
>for all form submissions. The character set utf-8 isn't available in MSSQL,
> if you want to work in UTF-8 you have to store the data in fields with a
> Unicode data type (nvarchar etc) and convert to and from UTF-8 on the
> way in and out. The way I work in an ASP/SQL Server environment is
> pretty simple...

Understood, you bet. But it's really hard to write it all down without those Martian characters.

By: bobo123

bobo123 — Wed, 26 Nov 2003 22:46:28 -0800

私は漢字を好む。

Character entities work fine ( for example: 日本 = 日本) but it's sort of a pain to figure out what code goes with what character. And when you hit preview it wipes all your codes from the entry window so you have to type them in again.

By: punishinglemur

punishinglemur — Sun, 30 Nov 2003 15:11:25 -0800

????????.

By: punishinglemur

punishinglemur — Sun, 30 Nov 2003 15:11:44 -0800

??
日本