Subject: [sup-talk] Unicode

From: [email-masked] (Dan Sully)
Date: Wed, 25 Jul 2007 08:34:29 -0700

I know Unicode support in Ruby is.. lacking. But is there anything I can do to
prevent the dreaded unicode question marks (often followed by a tilde), which
totally destroy the curses layout. 

I see that iconv is being used in sup, but I'm not sure when/why.

Thanks.

-D
--
<dsully> please describe web 2.0 to me in 2 sentences or less.
<jwb> you make all the content. they keep all the revenue.

-- 

-D
--
<dsully> please describe web 2.0 to me in 2 sentences or less.
<jwb> you make all the content. they keep all the revenue.

---

From: [email-masked] (William Morgan)
Date: Wed, 25 Jul 2007 10:42:38 -0700

Excerpts from Dan Sully's message of Wed Jul 25 08:34:29 -0700 2007:
> I know Unicode support in Ruby is.. lacking. But is there anything I
> can do to prevent the dreaded unicode question marks (often followed
> by a tilde), which totally destroy the curses layout. 

Here's what has worked for me:

1. Use gnome-terminal
2. set LANG=en_US.UTF-8

Then, somehow, magically, using ncurses to display utf8 characters just
works. Well, I haven't tried wide characters, but accented characters
work.

> I see that iconv is being used in sup, but I'm not sure when/why.

Sup uses iconv to convert messages (and individual headers) marked as
in specific encodings to your local encoding, as determined by LC_CTYPE
and LANG environment variables.

What *doesn't* work right now is calculating the lengths of unicode
strings, so the display is a little screwed up for characters that take
more than one utf-8 byte. But still usable.

-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Dan Sully)
Date: Wed, 25 Jul 2007 10:54:23 -0700

> Here's what has worked for me:

I'm using LANG, but with xterm-color (via Terminal.app to a Ubuntu box). That
setup works fine for Mutt, fyi.

I'll wait for another full wide subject line to come in, but the other ones
have been Windows encoded quotes and the like.
 
> more than one utf-8 byte. But still usable.

Ok, that would make sense.

-- 

-D

---

From: [email-masked] (William Morgan)
Date: Wed, 25 Jul 2007 11:50:47 -0700

Excerpts from Dan Sully's message of Wed Jul 25 10:54:23 -0700 2007:
> I'm using LANG, but with xterm-color (via Terminal.app to a Ubuntu
> box). That setup works fine for Mutt, fyi.

If you cat out a utf-8 file, does the terminal display the right thing?
If so, I would think that would work for Sup... at least under my
current mental model where ncurses doesn't do anything except put out
raw bytes which the terminal interprets.

> > strings, so the display is a little screwed up for characters that
> > take more than one utf-8 byte. But still usable.
> 

And in fact it's actually only a problem in thread-index-mode when
someone's name is a rfc2047-encoded header. In every other case there's
no character alignment issue. :)

---

From: [email-masked] (Dan Sully)
Date: Wed, 25 Jul 2007 12:00:24 -0700

> raw bytes which the terminal interprets.

Yep, it displays the right thing. I'll try and get you more data.

-- 

-D