Characters turning into question marks.

August 27, 2002 3:23 PM

Can any one see what is behind the box? (?)
I can't. I have tried every browser I have on two platforms, OS X and WinXP. Is this case for most people, and if so, why do people keep posting like that?
posted by Steve_at_Linnwood to bugs at 3:23 PM (26 comments total)

I have tried:
Mozilla 1.1 OS X - Filled in box
IE 5.2 OSX - Filled in Box
Chimera 0.4 - Filled in Box
Mozilla 1.1 WinXP - Question Mark
IE 6 WinXP - Outlined box

I find this odd, that people keep posting this way, when a majority, I would assume, can not see the "proper" punctuation they intend to use anyway. Is there a browser that does display this correctly?
posted by Steve_at_Linnwood at 3:26 PM on August 27, 2002

Damn Mozilla turns it into a ? mark, I can't post a box when I try!

You know what I am talking about.
posted by Steve_at_Linnwood at 3:28 PM on August 27, 2002

These characters show up correctly when previewed only to get screwed up when posted. See this for more detail.
posted by timeistight at 3:31 PM on August 27, 2002

I understand that, but people are going out of there way to use character codes that are known not to work. Why do they continue to do this? The boxes are more annoying than the improper usage of punctuation.
posted by Steve_at_Linnwood at 3:36 PM on August 27, 2002

Don't spell check or if you do, erase and re-write.
Yes sounds like old school, yet it works, no boxes, or question marks. I guess using Fortran in the 80's prepared me for something, besides y2k and the question why?, marked in a box. This is a double post every other week it seems since the beginnig of this problem. What am I saying as soon as this leaves the front page someone else will post it..........
posted by thomcatspike at 3:51 PM on August 27, 2002

They don't know Steve. It's like writing "Steve" and seeing it in a preview just fine, but when posted it becomes "$t3v3."

It's a non-trivial problem with a new server's database connector that no one in the ColdFusion developer community has seemed to solve. I'll keep looking around for a fix.
posted by mathowie at 4:27 PM on August 27, 2002

I find if I don't cut and paste, import from Word or Notepad or anywhere - or rewrite all characters if I do - there's no problem. Though sometimes it's easy to forget.
posted by MiguelCardoso at 4:31 PM on August 27, 2002

By default, Microsoft Word automatically replaces quotation marks with smart quotes, -- with proper emdashes, ... with ellipsis, TM with the trademark symbol, and so on. Every document I receive at work must be pasted into Notepad and subjected to half a dozen Find-Replaces before I can code the HTML formatting. Depending on your version of Arial, there's no way to tell the chars are illegal.

Thank Microsoft/AutoComplete for this wonderful feature.
posted by Danelope at 6:45 PM on August 27, 2002

Well, you could just use a plain text editor like Notepad, Emacs, UltraEdit (my favorite), or Dana...
posted by bshort at 8:40 PM on August 27, 2002

Danelope: Its easy to change that:

In Open Office goto Tools>AutoCorrect/AutoFormat.

In MS Office goto Tools>AutoCorrect
posted by skallas at 12:36 AM on August 28, 2002

How many times has this been brough up on MetaTalk, now? 3-4 threads to itself? Perhaps it's time for a bit of social engineering until the database engineering can catch-up.
posted by nathan_teske at 3:24 AM on August 28, 2002

skallas: Danelope: Its easy to change that

Not once the document or post is already written, however. In my case, I'd have to disable it on every machine at the office, and ask everyone who submits documents to my company to disable it as well.

In other words, I'm damned to a life of search-and-replace.
posted by Danelope at 6:46 AM on August 28, 2002

Danelope -- it's a bit out of date, but the demoronizer might be useful to you...

Come to think, a reasonable and not-too-difficult-to-code workaround for the coldfusion problem would be to pass new comments through a quick search-and-replace before storage, just to whack the most common problem characters (just getting rid of "smart" quotes and em-dashes would get you 90% of the way there.) (I'm probably not the first one to suggest this, though, am I?)
posted by ook at 10:57 AM on August 28, 2002

ook: I actually posted a link to demoronizer to my weblog about a year ago, but had completely forgotten about it since then. Thanks for reminding me!
posted by Danelope at 11:18 AM on August 28, 2002

actually ook, there is a dearth of code libraries for coldfusion to do just that, but they were written for the previous version of the server software. I haven't seen anything updated for the current problems.
posted by mathowie at 11:38 AM on August 28, 2002

Danelope, I feel your pain.

mathowie, this is incredibly hacky, and wouldn't do anything for stuff already in the database, but would at least clean up future input a bit before it ever reaches CF:

function debox() { // js called from form onSubmit
var it = document.mefi.comments.value;
if (it.replace) {
it = it.replace(/[”“]/g, '"');
it = it.replace(/[‘’]/g, "'");
it = it.replace(/pancake/g, "pony");
// etc...
document.mefi.comments.value = it;
}
return true;
}


posted by ook at 2:28 PM on August 28, 2002

...that is, it'd work if I could figure out how to type the bad characters into the regexp without, uh, typing the bad characters...

never mind then...
posted by ook at 2:30 PM on August 28, 2002

ook, here's a general purpose function that'd replace all high ASCII characters with an nbsp. The only problem is it'd also strip characters with cedillas and circumflexes and umlauts and the like. And do nothing with pancakes.
//
function stripHigh() {
var input = document.mefi.comment.value
var output = "";
var inChar;
for (var i = 0; i < input.length; i++) {
inChar = input.charCodeAt(i);
output += (inChar > 127) ? " " : String.fromCharCode(inChar);
}
document.mefi.comment.value = output
return true
}
posted by TimeFactor at 3:19 PM on August 28, 2002

Well there you go, then. What do we need umlauts for, anyway?

Hey, look at that -- our user numbers went away... however will I know who to suck up to, anymore?
posted by ook at 9:08 PM on August 28, 2002

mathowie: s/dearth/wealth
posted by dhartung at 9:13 PM on August 28, 2002

Just to clarify, I'm not insane. During preview, if you roll over a username, you get the username.mefi/whatever url; otherwise you get the number. Which I never noticed before until just now, which is why that half-witty little side comment I just made makes more sense than you think it does. And yes, I do realize I'm just digging myself in even deeper here, so I'll stop now. But I still see no use for umlauts. And I'll stand by that. And I still don't know who to suck up to, anymore.
posted by ook at 9:20 PM on August 28, 2002

Gee, I just previewed and then posted a post full of high-bit entities — and it worked fine! Is this fixed? “Here are some ‘entities’ — just for laffs!” Does anyone see boxes there?
posted by nicwolff at 10:53 PM on August 28, 2002

Oh, FYI, i'm using Mac IE 5.2 on OS X 10.1.5.
posted by nicwolff at 10:53 PM on August 28, 2002

nicwolff --- the problem aren't html entities. They're un-entitied hi-bit characters being passed off by the script. — Æ things like that
posted by nathan_teske at 1:19 AM on August 29, 2002

Odd --- My option-shift-hyphen em dash — got snarggled but my option-shift-' AE ligature Æ didn't. IE 5.2.1 OS X 10.2
posted by nathan_teske at 1:22 AM on August 29, 2002

It is learning.
posted by skallas at 1:30 AM on August 29, 2002