Subject: Re: [sup-devel] Arch utf8 vs UTF-8 fix and wide character support

From: William Morgan <[email-masked]>
Date: Tue, 11 May 2010 15:19:43 -0400

Reformatted excerpts from Matti Eiden's message of 2010-05-11:
> I don't know what is the opinion of other sup users, whether adding a
> new dependency (to terminfo) is desirable, as the current list of
> dependencies is already rather high.

I'm working on a gem that provides both wcswidth functionality and the
ability to slice a string at specific display widths, both of which, I
think, will be required to function entirely.

You can try it with 'gem install console'. It's 1.9-only for now.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Rich Lane <[email-masked]>
Date: Fri, 07 May 2010 12:46:20 -0400

Excerpts from Matti Eiden's message of 2010-05-06 14:02:46 -0400:
> Hey folks,
> 
> I've been experimenting with sup for the past few days, and of course,
> I love it. Firstly I had some trouble with getting unicode display
> going. This problem was already described in an old post on this
> mailing list:
> 
> http://rubyforge.org/pipermail/sup-devel/2010-March/000522.html
> 
> So Arch Linux defines encoding as utf8, but Iconv requires it to be
> UTF-8. I would say this is a bug in Arch Linux for not following
> standards, but anyway, I fixed it with the little modification to
> sup.rb:
> 
> ## determine encoding and character set
> $encoding = Locale.current.charset
> $encoding = "UTF-8" if $encoding == "utf8"

I've applied this fix, thanks.

> Then about wide character support. And I mean really wide. Like CJK
> characters. Scandics (ä,ö,å) and other European accent characters work
> nicely, as we all who are concerned probably know. These characters
> have a byte length of 2 and unicode length of 1.
> 
> However, take an example of the following two-character Korean word
> (byte length of such single character is 3 instead of 2!)
> 
> http://www.kotiposti.net/eiden/soulbound/hellovim.png (looking good in vim)
> http://www.kotiposti.net/eiden/soulbound/hellosup.png (sup lost 2
> characters (or bytes) from the line that has the Korean word)
> 
> It seems that for every Korean character with a byte length of 3, one
> byte is lost from the end of the line. In the above example, two bytes
> are missing in sup, as there are two Korean characters on the same
> line.
> 
> If the line consist of a single Korean character, nothing appears in
> sup (last byte out of three is missing?).
> If the line consist of two Korean characters, last character is
> missing (last two bytes out of six are missing?).
> etc.
> 
> Some sort of miscalculation somewhere is causing this, perhaps
> assuming that unicode characters always have a byte length of 2? Can
> anybody with Ruby skills take a look on this?

It's actually the multiple screen cells that causes problems, not
multiple bytes [1]. Sup currently thinks all characters are 1 cell wide.
The right thing is probably a C extension that uses wcswidth.

[1] http://mid.gmane.org/1264629880-sup-9232%40zyrg.net
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Thu, 13 May 2010 08:33:50 -0400

Reformatted excerpts from Matti Eiden's message of 2010-05-11:
> Oh right, splitting. Yes right, makes sense. I tried your
> console/string, seems good, except the display_split ignores the
> padding request? Or did I understand this feature wrongly? I mean, it
> slices the string exactly where the end offset is, not by the nearest
> " ", space?

The padding only comes into play when you ask it to slice a string in
the middle of a multi-column character. It will then pad the
left/right/both ends of the string to make the total display width of
the returned string what you asked for.

> With quick testing for resizing the window with different kind of test
> emails, I see no lost characters or text corruption.

Awesome. I'll see what I can do about 1.8-compatibility.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Matti Eiden <[email-masked]>
Date: Thu, 6 May 2010 21:02:46 +0300

Hey folks,

Thanks,
Matti

---

From: Matti Eiden <[email-masked]>
Date: Wed, 12 May 2010 00:51:20 +0300

Well anyway, I showed it inside sup, seems to be working nicely.

Here's what I did to get it to work, if anybody's interested. I'm in a
rush to work, so there may be mistakes. I tried to check that
everything works.

Summary:
- buffer.rb is patched to slice all strings according to @width, this
fixes issues in inbox-mode when email subjects have wide characters.
Old "hacks" were removed.
- utils.rb is patched to wrap using display_slice and then looking for
nearest space. if no space is found, it uses simply the original
output of display_slice. display_length function defaults to the
display_width

Nice, thanks.
--- buffer-old.rb	2010-05-12 00:42:50.501278238 +0300
+++ buffer.rb	2010-05-12 00:42:37.711280439 +0300
@@ -1,5 +1,6 @@
 require 'etc'
 require 'thread'
+require 'console/string'
 
 begin
   require 'ncursesw'
@@ -129,10 +130,8 @@
     @w.attrset Colormap.color_for(opts[:color] || :none, opts[:highlight])
     s ||= ""
     maxl = @width - x # maximum display width width
-    stringl = maxl    # string "length"
-    ## the next horribleness is thanks to ruby's lack of widechar support
-    stringl += 1 while stringl < s.length && s[0 ... stringl].display_length < maxl
-    @w.mvaddstr y, x, s[0 ... stringl]
+    s = s.display_slice(0,maxl,"")
+    @w.mvaddstr y, x, s
     unless opts[:no_fill]
       l = s.display_length
       unless l >= maxl
--- util-old.rb	2010-05-11 21:38:55.736596584 +0300
+++ util.rb	2010-05-12 00:33:16.128001053 +0300
@@ -3,6 +3,7 @@
 require 'mime/types'
 require 'pathname'
 require 'set'
+require 'console/string'
 ## time for some monkeypatching!
 class Lockfile
   def gen_lock_id
@@ -177,16 +178,12 @@
 end
 
 class String
-  ## nasty multibyte hack for ruby 1.8. if it's utf-8, split into chars using
-  ## the utf8 regex and count those. otherwise, use the byte length.
+
   def display_length
-    if RUBY_VERSION < '1.9.1' && ($encoding == "UTF-8" || $encoding == "utf8")
-      scan(/./u).size
-    else
-      size
-    end
+    display_width
   end
 
+
   def camel_to_hyphy
     self.gsub(/([a-z])([A-Z0-9])/, '\1-\2').downcase
   end
@@ -270,14 +267,17 @@
   def wrap len
     ret = []
     s = self
-    while s.length > len
-      cut = s[0 ... len].rindex(/\s/)
-      if cut
-        ret << s[0 ... cut]
-        s = s[(cut + 1) .. -1]
+    while s.display_width > len
+      cut = s.display_slice(0,len," ")
+      # find the last space, since display slices it precisely
+      space = cut.rindex(/\s/)
+      space = cut.size unless space #No spaces?
+      cut = s[0 ... space]
+      ret << cut
+      if space != cut.size #+1 to kill the space in the beginning of next line
+        s = s[(cut.size + 1) .. -1]
       else
-        ret << s[0 ... len]
-        s = s[len .. -1]
+        s = s[cut.size .. -1]
       end
     end
     ret << s
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Matti Eiden <[email-masked]>
Date: Tue, 11 May 2010 21:50:17 +0300

> [1] http://mid.gmane.org/1264629880-sup-9232%40zyrg.net

So okay, I sent my previous answer accidentally only to Rich. In this
mail I mentioned a Ruby library called terminfo (
http://www.a-k-r.org/ruby-terminfo/ ), which contains a function
wcswidth.

I downloaded terminfo and installed it, and edited lib/sup/util.rb slightly.
1) Added of course "require 'terminfo'" on the top.
2) Modified the display_length function of "nasty multibyte hack" to
use TermInfo.wcswidth instead of native "size"

Results: Everything seems to work now.

I don't know what is the opinion of other sup users, whether adding a
new dependency (to terminfo) is desirable, as the current list of
dependencies is already rather high.. Discuss. If somebody with some C
skills knows how to move that wcswidth function to the ruby-ncursesw
(Rich? :D) would that be a more favourable option?

Here's the actual patch of what I did to keep it clear (I notice there
has been an earlier utf8 patch here, for pre Ruby 1.9.1 versions. I
don't know how many of you use some earlier Ruby version still, of
course this would also solve the "nasty" utf scan patch.):

--- util-old.rb	2010-05-11 21:38:55.736596584 +0300
+++ util.rb	2010-05-11 21:36:12.653281044 +0300
@@ -3,6 +3,7 @@
 require 'mime/types'
 require 'pathname'
 require 'set'
+require 'terminfo'
 ## time for some monkeypatching!
 class Lockfile
   def gen_lock_id
@@ -183,7 +184,7 @@
     if RUBY_VERSION < '1.9.1' && ($encoding == "UTF-8" || $encoding == "utf8")
       scan(/./u).size
     else
-      size
+      TermInfo.wcswidth(self)
     end
   end
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel