Subject: [sup-talk] Character encoding when displaying quoted-printable messages

From: [email-masked] (Jim Cheetham)
Date: Thu, 18 Jun 2009 09:39:10 +1200

Hi, I've been looking at sup for only a day, as I'm trying to find a
decent email client. I migrated from mutt to gmail, but now it's time
to move away from gmail because their client is not smart enough.

Sup has passed my number 1 feature test; I can easily read messages
while keeping multiple unsent messages available. Mutt does that, but
it's not easy to see what messages you are currently working on.

However, I'm coming across some character decoding issues with
quoted-printable that are making some messages hard to read. Any
advice on how to improve things would be welcome ... In the example
below, a NBSP entity is showing up as the characters "M-BM-"

I'm on Ubuntu 9.04, with the current master from git. Sup is running
in GNOME Terminal 2.26.0, with UTF-8 encoding, although unsurprisingly
switching to ISO-8859-1 makes no difference. $TERM=xterm (or 'screen'.
vt100 and vt220 cause sup to halt at startup).  This is the text that
gets displayed for a sample message sent from Yahoo! mail ...

Visible in sup thread-view-mode:
videos for specific hints regarding the gestures.M-BM-  Do you have a decent
recording device?M-BM-  Have you seen what's available?M-BM-  Do you have
editing software?

Message source:
X-Mailer: YahooMailClassic/5.4.12 YahooMailWebService/0.7.289.15
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1465842634-1245251490=:14771"
...
--0-1465842634-1245251490=:14771
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
...
more videos for specific hints regarding the gestures.=A0 Do you have a dec=
ent video recording device?=A0 Have you seen what's available?=A0 Do you ha=
ve any video editing software?

Displayed in Gmail client:
I was thinking about cleaning up the videos on YouTube, and perhaps
adding more videos for specific hints regarding the gestures.  Do you
have a decent video recording device?  Have you seen what's available?
 Do you have any video editing software?

HTML source when viewing the MIME attached message locally:
I was thinking about cleaning up the videos on YouTube, and perhaps
adding more videos for specific hints regarding the gestures.&nbsp; Do
you have a decent video recording device?&nbsp; Have you seen what's
available?&nbsp; Do you have any video editing software?<br><br>[1]

---

From: [email-masked] (Jim Cheetham)
Date: Wed, 1 Jul 2009 17:20:10 +1200

> below, a NBSP entity is showing up as the characters "M-BM-"

Nicely fixed by using the hack ncursesw branch, by the way.
http://sup.rubyforge.org/wiki/wiki.pl?UTF8

-jim

---

From: [email-masked] (Ingmar Vanhassel)
Date: Wed, 01 Jul 2009 10:38:36 +0200

> http://sup.rubyforge.org/wiki/wiki.pl?UTF8

Has anyone noticed that newer versions of ncurses-ruby plus that
ncursesw hack [1] (to link to ncurses with widechar support) breaks search?

When I search for 'foo', sup launches a search for a bunch of weird
characters, not for 'foo', which obviously returns nothing useful.

I can trigger this with ncurses-ruby 1.2.2, 1.2.3. Version 1.1 seems to
be unaffected.

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=477366#52
-- 
Exherbo KDE, X.org maintainer

---

From: [email-masked] (William Morgan)
Date: Wed, 01 Jul 2009 08:28:46 -0700

> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=477366#52

Interesting thread. I disagree with the conclusion that a separate
libncursesw-ruby package should be created. Maybe I'll respond.

The analysis of Sup is wrong. He says: "Looking at the source code of
the mailer I'd say that it is not really suited for UTF-8 encoded
strings yet, as it still assumes that the length of a string in bytes is
equal to the number of characters in the string." This is not really
true. What Sup assumes is that String#length gives the length of the
string in characters. This is obviously false in Ruby 1.8 (without
monkeypatching), but it is true in Ruby 1.9.

What *does* need to happen for Sup to really, actually, correctly
display non-ASCII characters, besides having a ncurses library that's
linked against ncursesw, is to be able to call the 'wcwidth' and
'wcswidth' functions. The current #display_length function is a hack
that's broken for e.g. Chinese characters.

One way to do this would be to use dl/import like we do for setlocale().
I've played around with this but haven't really gotten it to work. If
someone can get any further than the below, please let me know:

  require 'dl/import'
  module LibC
    extend DL.const_defined?(:Importer) ? DL::Importer : DL::Importable
    dlload "libc.so.6"
    extern "void setlocale(int, const char *)"
    extern "int wcwidth(int)"
    extern "int wcswidth(const int*, int)"
  end

That "works", as in doesn't throw any exceptions, but then calling
LibC.wcswidth doesn't return the right thing, presumably because the
Ruby string isn't being converted into an array of ints, or something. I
don't really know how this works.

> ncursesw hack [1] (to link to ncurses with widechar support) breaks

I don't have these newer versions available on my system yet (Ubuntu).

> characters, not for 'foo', which obviously returns nothing useful.

Disturbing.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Jörg-Hendrik Bach)
Date: Wed, 01 Jul 2009 17:42:04 +0200

> ncursesw hack [1] (to link to ncurses with widechar support) breaks search?

I can confirm the behaviour.

> characters, not for 'foo', which obviously returns nothing useful.

True. However, with the following workaround, you can get proper search results:
- search for foo -> garbled search, no proper results
- hit 'x' to destroy the result buffer
- hit '\' to search again.
- hit uparrow to go to previous search. backspace all the garbled characters there.
- repeat the uparrow + remove until it shows your last working search (or until there's no more previous search to go to)
- type foo again, hit enter: works.

Not that this is the type of thing you'd want to do for a simple search. Guess we'll have to wait for ruby 1.9?

cheers,
- J?rg-Hendrik